Beta testing and usability testing are crucial elements of the product development process. We wanted to learn more about the beta testing space, so we welcomed Luke Freiler, CEO of Centercode, to the Built Right podcast to talk about his journey and his insights as a CEO and Founder.
Luke takes us back 22 years to how he first got interested in the space and eventually became the CTO and then CEO of Centercode. We hear his thoughts on agile working, AI, building a framework into the testing stage, and what his company looks for during tests.
Keep reading for the top takeaways from this episode or tune in below.
The difference between QA and usability testing
How Luke looks at beta testing is beyond the general QA that products go through. They dig into the code and try to make sure everything is performing as expected to start with. But on the UX side, it’s more about the experience and how people use the product. Centercode’s approach is to look at how users interact with a product over a period of time, rather than basic UX testing, which is very observatory and less of a realistic test.
As Luke explains, this leads to quality issues that wouldn’t immediately become apparent in a lab test.
Beta testing is about more than hunting for bugs. While that’s an important part of it, Luke is really looking for three things when testing:
- Issues – is the product doing what it’s supposed to?
- Ideas – what could be improved?
- Praise – what do people love about the product?
Those three things together are paraphrased as “fix, improve, promote,” which is a key philosophy at Centercode.
The importance of having a framework
Centercode got to the point where it had a central team that ran one of the most diverse beta tests in the world. But one issue that Luke noticed is that there’s no framework for beta testing – everyone does it a little differently and is recreating it from the ground up. This means the opportunity cost is huge.
So, Luke started to think beyond the technology itself and more about the process. Centercode already has a lot of information about products and usability testing, and this led them to design a framework and certifications to try to improve the beta testing space.
Luke’s goal was to “get rid of the first thousand hours that every company was going to use to reinvent their own wheel.”
Centercode was agile in the early days, and Luke loved that you could finish a feature and get it into people’s hands quickly. Rather than doing a major release where half the features are done until they work on the other half, you could get it out sooner and develop as you go.
The main thing was that people are paying for something and want persistent value, and the agile way of working became a no-brainer for Luke.
However, it didn’t go well at first. The company and even the customers didn’t necessarily want releases and changes that often. So they eventually walked away from the agile approach on the customer front, but still practiced agile development internally.
However, over time, Centercode built its team up and built better project management structures in place to do rollouts more efficiently. Now, they’re back to doing agile development, and it’s been working well for the last two years.
AI is user–testing
There’s a lot of hype around AI in the product development world, and the beta testing arm of it is no different. After being impressed by the content that ChatGPT could produce, Luke said to his team, “AI is not going to replace you. Someone who uses AI will. So let’s be that person.”
Centercode is now experimenting with different ways it can use AI both for products and internally. There’s currently a backlog of great ideas for ways to use AI. However, Luke is quick to mention that AI is still in its early days. We’ve yet to see a mature AI product, so we can’t say what AI tools will be like in the future. But they need to be reliable and predictable, most of all.
However, it’s clear that AI is here to stay for the foreseeable future, and it’s important to experiment with it or get left behind.
For more insights into Centercode and beta testing as a whole, check out the full episode of the Built Right podcast.
[00:00:00] Matt Paige: Welcome Built Right listeners. Today we’re chatting with Luke Freiler, CEO of CenterCode, and Luke’s Journeys is a bit unique in the fact that he’s been running Center Code for the past 22 years, starting as CTO been moving into the position of CEO. And we’re gonna get into his journey and hear the type of stories you only get from a founder, like how he got into product, market fit, insights on pricing, go to market, and really everything in between.
[00:00:32] Matt Paige: But welcome to the show, Luke.
[00:00:33] Luke Freiler: Thank you sir.
[00:00:35] Matt Paige: Alrighty. So instead of like, you know, Luke, you’re telling us what Center Code does. I kind of want take us back 22 years, like go back to. When you first realized this problem existed, it’s a really cool story we were chatting about in advance. I want to get into that, um, and, you know, kind of take us there to kind of get into what Centercode does
[00:00:55] Luke Freiler: Absolutely. So I, I started my career from a, a pretty young age in tech. I, I started out at Samsung, uh, quickly moved with a, a big group of people over to Ericsson, which at the time was a a hundred year old, a hundred thousand person tech company. And at the time the internet was still pretty fresh and new and, and I was young, but excited and, and pretty invested in it.
[00:01:16] Luke Freiler: And, and specifically the thing that really lit me up was just the idea of, of usability and, and what eventually became user experience. You know, back then tech products were pretty much used by tech people and, and anything that broke that mold was, was pretty novel. So, uh, I, on the other hand had low patience despite being a, a techie person and, and kind of grew up supporting.
[00:01:37] Luke Freiler: You know, the school network and printers and whatnot, and, and always thought there should be a better way. So that’s kind of where my head was. And, and where I was at, Ericsson was running a, a web development team that handled any sort of web oriented functionality for the product that we were bringing to market.
[00:01:53] Luke Freiler: So firmware of the product itself, internal sites, public marketing sites, everything. So I had a group under me who were all, again, sort of young web developers and a product manager came to me and he said, Hey, I need you to run a beta test for our product. Hmm. And I, I kind of lit up and said, okay, well what does that mean?
[00:02:11] Luke Freiler: And, and he said, well, what is a beta test? It’s, yeah, it’s a, it’s a, it’s when you get a bunch of customers. And, and I was like, no, no, I, I know what a beta test is. I don’t know what that means at Ericsson, right? We, we have a process for everything. I have a four inch guide on how to use the logo. Um, there, there’s gotta be a path for me here so I don’t have to just create it.
[00:02:29] Luke Freiler: And he was like, no, man. Best I can tell there, there isn’t you. You’re gonna have to just figure it out. And I said, okay, can I outsource it? Can I go find a vendor? And he said, sure. And I said, okay, I’ll, I’ll do that. Give me a budget. So I started looking around, I started researching and, and I quickly realized that it, it was a gap in the market.
[00:02:43] Luke Freiler: It, it, what I discovered was this rare, necessary evil in business. Something that everybody sort of universally agrees is important is, is positive, but it’s hard. It, it’s, it’s challenging the, the idea of. I’m going to get a unfinished product and, and give it to a bunch of strangers and, and expect meaningful feedback that I can transform into a successful product.
[00:03:06] Luke Freiler: It’s just a tall order. So at that point, I, I saw that opportunity and, and like I said, I was already kind of borderline obsessed with the, the future of usability and user experience and technology being more useful and, and it all just kind of came together. And from there, I, I decided after talking to many, many people and asking everyone I knew and everyone within their networks how they did this, I, I kind of realized that the challenge was, it was something that happened.
[00:03:31] Luke Freiler: Episodically e every once in a while somebody would run a beta test for a product. Yeah. Uh, back then everything was waterfall development, so it was a good couple years between products. Product managers were heads down for a very long period of time. They would run a big beta test for eight or 12 weeks At the end of that, But then there’d be another two years before they did it again.
[00:03:49] Luke Freiler: So as a result, they kind of forgot everything they learned. And that led me to a, a lot of different paths that that all turned out to be. Yeah, there, there’s an opportunity here. There’s there’s an amazing business opportunity that not only fits what I’m passionate about, which is incredible, but also it’s sort of universally positive.
[00:04:06] Luke Freiler: The, the idea of taking a company and helping introduce them to their customers and then, Sort of orchestrating the, the communication between them and, and helping them ultimately just produce a better product for those customers is just a universal win for everyone involved. So I never expected to spend most of my career in any one place.
[00:04:25] Luke Freiler: That certainly wasn’t something I I set out to do. I also didn’t ever expect to be a c e O. Um, I just kind of fell into something I loved and, and invested heavily in it, you know, of my own personal time. And I, I don’t see myself going anywhere. I, I absolutely love it and we get to work with many of the greatest brands ever.
[00:04:41] Luke Freiler: Uh, to bring products to, to billions of people. So it’s awesome.
[00:04:45] Matt Paige: Yeah, and it’s a lesson in, we talk about the, the built right method at Hatch Works and it’s centered in building something that’s valuable for the customer, viable for the business, and feasible like the Marty Kagan kind of school of thought.
[00:04:58] Matt Paige: And you know, we’ll see clients a lot of the times they’ll come to us with an idea, but they’re kind of missing like, what’s the problem behind it? And you know, so that’s a key piece that often. It gets overlooked. But I, I wanna get into, you mentioned this usability testing. A lot of people when they think of testing, they think of just pure kind of QA testing within a solution, which is different though than when you talk about usability beta testing.
[00:05:22] Matt Paige: Uh, for those that may maybe not know that kind of difference there, like what, what’s the difference between those and kind of purpose of why you’re doing these usability tests?
[00:05:29] Luke Freiler: So, so traditional just QA testing is, is pretty contained to professionals doing it. Now there are sort of crowdsourced options nowadays, but ultimately you have people who are walking through test scripts and doing very explicit things to basically make sure the product is performing as it was expected to.
[00:05:48] Luke Freiler: Um, on the other side of that, you have UX testing where it’s much more focused on the experience and do people understand it? Is it producing an experience that that actually gives them the outcome they want? Um, what we do is a little bit different and the sense that it’s all about real users. The, the people who actually have the problem, the product solving that will buy the product eventually.
[00:06:09] Luke Freiler: In their real environments and then over a period of time. So whereas a UX test is often, um, observatory, we’re, we’re watching somebody do something and taking notes and taking action from that. And it’s very qualitative in that sense. Um, what we’re doing is bringing something a little broader and it’s more about, okay, how do these real people adopt the product over time?
[00:06:28] Luke Freiler: How do they use the product in their real environments, which then naturally leads to a lot of quality issues that you’re not necessarily going to find in a log, uh, a lab and many people do immediately recognize beta as a bug hunt, and that’s certainly a critical part of it because the real world does just uncover an enormous amount of situations and scenarios that, that an, a business environment wouldn’t.
[00:06:49] Luke Freiler: Um, but it goes far beyond that, and from our perspective, We’re always looking for three things and, and we’re looking for issues in the product. You know, what’s it not doing that, that it, that it is supposed to be doing? Uh, what’s not behaving as the customer expects it to. We’re looking for ideas. What could be improved in the product, either small improvements all the way to features that could end up in future versions or, or what competitors are doing.
[00:07:11] Luke Freiler: That might be an expectation they didn’t see coming. And then the last thing we’re looking for is, is praise. What do people love about the product? What is it that that really stands out to them? And what could be promoted to grow the audience? And. Those three things together, issues, ideas, and praise. You know, we sort of paraphrase it as fix, improve, promote, um, that’s what we’re going for.
[00:07:29] Luke Freiler: Mm-hmm. We’re trying to look at this product in, in that real world and say, okay, how can we maximize the success of this product? And, and how can we use a, an audience of enthusiastic volunteers to make that happen? Yeah. And you know, there’s,
[00:07:41] Matt Paige: I wonder if this is another one or if it fits into like the, the idea improved category, but we’ll see a lot of times when, you know, a product actually gets into the hands of end users.
[00:07:50] Matt Paige: They start using the tool in unintended ways and the initial reaction’s like, oh man, that’s the wrong way. You’re doing it wrong. But they may unearth like whole new ideas Yes. With how to use the solution and get value from the solution. Right. Oh
[00:08:03] Luke Freiler: yeah. It, it that, that promote side and, and the praise is where you find the things about your product, you know, that, that happen, that, that are just happy accidents.
[00:08:12] Luke Freiler: Um mm-hmm. And, and that happens every time. I mean, virtually every time there are gonna be things about your product. That are problematic that you didn’t see coming, that’s the most obvious thing. But to sort of balance out the morale, uh, you’re gonna see just as many things that, that are new opportunities to, to leverage something you did in that product that you, you know, sort of emergently came out you didn’t even see coming.
[00:08:31] Luke Freiler: Mm-hmm.
[00:08:32] Matt Paige: Take us back to, uh, your first deal. This is such a, a cool story. Your first deal, your first conversation when you got this like, idea of a product you’re building. Yeah, I think it’s such a good lesson for folks in terms of really value, I wanna say pricing, but it’s, it’s all about value.
[00:08:48] Luke Freiler: I, I spent, I, I’ve spent decades on unpacking this deal and, and trying to figure out, you know, what, what to do with it.
[00:08:54] Luke Freiler: But yeah, it’s, it’s, it is a great story. So, when I started this company, uh, I was an engineer at heart and I, I brought together a few people from Erickson with me. We all left and went and started this. And I was leading the tech side. As I said, I, I wasn’t a A C E O at the time, but I was the product guy.
[00:09:09] Luke Freiler: It was my vision, my idea, and I, my first idea for this company was that this was gonna be a platform that we were gonna sell online for a couple hundred dollars a month, and it was gonna be sort of SaaS before SaaS and, and that sort of self acquisition before product-led growth was a thing. This is, again, going back 20 plus years.
[00:09:26] Luke Freiler: But that’s how I thought. I, I wasn’t a sales guy. I didn’t want to be a sales guy on the phone. I, I just wanted to, to sell online and, and be frictionless so I could go focus on building something cool. And one of the first calls we got was from Sun Microsystems, who at the time was an absolutely enormous entity.
[00:09:42] Luke Freiler: They made the Solaris operating system, which was kinda the heart of business. Uh, big Deal, eventually acquired by Oracle. But at that point in time, they, they were one of the biggest names in tech and, and they called us and, and they wanted to inquire about the product. And I, I was in shock, like, I, I know how to handle this.
[00:09:59] Luke Freiler: I, I basically was a kid and I, I brought someone on the call who I had worked with at, at, at Samsung, initially, a guy named Scott. Mm-hmm. And Scott had spent 30 years at i b m in a, you know, growing to very, very senior positions running B two B sales at I B M. And I basically asked him to come be the adult on the call.
[00:10:17] Luke Freiler: Uh, I, I would be there to answer any of the product questions, but as far as sort of orchestrating this call, I, I really needed his help. And, and he did that as a favor to me. He was great. And ahead of this, we hadn’t talked about what the product would cost for me. It, it was gonna be a couple hundred bucks a month.
[00:10:33] Luke Freiler: It was gonna be self-serve, credit card, all that kinda stuff. And that wasn’t his world, but we didn’t really talk about it because we didn’t think the price would come up on the first call. My understanding of, yeah, those large sort of B two B engagements were that they had numerous fact finding calls and so on, and it was gonna take a while to even get there.
[00:10:48] Luke Freiler: So we get on the call, we, we show them the product, which wasn’t done. It was far from done in fact, and we were bootstrapping this company and, and kind of working it out on, on, on our own time. And I started trying to throw every objection possible at, at these guys. And I was saying like, Hey guys, it’s not done yet.
[00:11:04] Luke Freiler: And their response was, well, that’s great. We can help shape the roadmap. And you know, I I, I felt like they must have thought hard, playing hard to get was my sales strategy when in reality I was just scared to death that, that they were gonna overwhelm us and, and we were gonna disappoint them and, and, you know, it was gonna be crushing.
[00:11:19] Luke Freiler: So anyway, we show them the product, they love it. Um, we, we get to the point where they eventually said, just flat out, you know, what, what does it cost? And I’m sitting there deer in headlights. I don’t know what to say. And, and Scott and I hadn’t talked about this, and Scott just immediately blurts out, well, it’s $300,000 and 60,000 a year in, in perpetuity for support.
[00:11:38] Luke Freiler: And they flat out said, that sounds reasonable. We can do that. And I’m like, we get off the phone and, and I’m like, blown away. Like, I just found my funding, you know, from my perspective. And yeah. And Scott’s like, damn, it should have said 500. Um, and, and I’m sitting here going like, yeah, man, I just said $200.
[00:11:54] Luke Freiler: So, yeah. Thank you. I guess. And, and we sold it to them and, and they ended up a customer for a very, very long time. It was, it was a great experience, great relationship. And, and they did in fact, basically, you know, without knowing it, fund the company for that year. And, and on top of that, like even it, it even went further when we said to them like, Hey, we’re, we’re not on your platform.
[00:12:14] Luke Freiler: They were like, well, great, we’ll just send you a whole bunch of hardware. And they funded all of our infrastructure. As, as test equipment. Wow. Like tens of thousands of dollars, which at the time was amazing for us. We were, you know, living on 79 cent tacos. So it was, it was an incredible first experience and it did sort of reshape the business in, in many ways moving forward.
[00:12:34] Luke Freiler: Yeah, it’s such a good story
[00:12:35] Matt Paige: just on the, the concept of perceived value. Right. And, and that’s the key thing is kind of getting to, um, A sense, like how do you even get to a sense of what that is without having this serendipitous moment like you did, where you had the, uh, the guy in
[00:12:52] Luke Freiler: the room that did this?
[00:12:54] Luke Freiler: It, it’s, it’s fascinating because I, I like that story because it’s, it’s pretty wild and it’s totally true when I was there. Mm-hmm. But it’s actually not all good. And, and I, I, once I, I spoke at a conference about software pricing once, and the guy that spoke next to me was a guy named Rick Nucci, who was amazing.
[00:13:12] Luke Freiler: Mm-hmm. He’s, uh, the founder of, um, Boomie that sold to Dell for a bunch of money. Very successful guy runs a company called Guru now, I believe. And I remember him standing up before I spoke and, and said, uh, something along the lines of, you know, we’ve spent as much time on our pricing as we have our product.
[00:13:26] Luke Freiler: I like literally laughed out loud in, in the audience. And the reason was because what I was writing or talking about to that group that day, um, this was some sort of SaaS conference is, uh, sort of our journey of pricing and how every price we ever had, like the pricing we have today, could not look more different than that first price.
[00:13:45] Luke Freiler: It’s actually in a lot of ways, closer to my first vision for the company, ironically. But every price we had for about a decade and a half after that. Um, had some roots in that original price that we made up on the fly, and the reason was because that price was accepted, Scott basically threw down the gauntlet and said, look man, this is what the market will bear.
[00:14:05] Luke Freiler: They accepted it. This is what’s gonna happen. Mm-hmm. And what we recognized later is that they were actually a unique company in a unique, in a unique situation. What, what they had was an operating system to test. And if you think about it, operating systems were basically the first agile developed products in tech because you couldn’t just sit on your hands for two years.
[00:14:26] Luke Freiler: In between updates, there is security problems. Mm-hmm. There’s compatibility issues constantly coming out. So people have been patching, uh, operating systems frequently, forever. And where I’m getting at with this is, is they had a unique need that we didn’t understand was a unique need. Yet we pinned that as the market.
[00:14:43] Luke Freiler: And for years after, and I used to have this, this deck that kind of beautifully outlined this, every price that we came up with from there, every new model we came up with was somehow traced back to that original price. So if it was first 300,000 with 60,000 in support, I would make the argument of, oh, let’s get it down to.
[00:15:00] Luke Freiler: You know, 120,000 a year, but now it’s under 20,000 a year. So over a certain course, it’s the same amount of money. Then we would get and say, okay, well our average customer has six seats, uh, six full-time users who live in this product, so let’s sell it for $20,000 a seat. I could actually trace back like a decade and a half of pricing strategies that were all hooked to this idea of we don’t wanna give up or cannibalize that very first customer.
[00:15:24] Luke Freiler: And the problem with that thinking is we didn’t accept for the longest time that that first customer had a very unique model. And again, the irony is they were just early to agile and because they had agile development, they had a continuous need. They could justify building a team around this in one place, and therefore they could justify the budget to support that team.
[00:15:44] Luke Freiler: Whereas for the next decade, plus most companies were putting this on individual product managers like we had at Ericsson, which meant they were only actually running a beta test every couple years. And when you have that, you don’t have, when you’re decentralized in that form, you don’t have a centralized team, a centralized process.
[00:16:00] Luke Freiler: So you don’t have tools to support that centralized process. You don’t have a budget for those tools. And that actually held us back again, we were bootstrapped, so things like awareness weren’t something we could invest heavily in. And as a result, that kind of held us back sticking to those guns. And it was only in recent years after I took over c e o and, and wanted to get a lot more aggressive with how we approached the market and rethink what our go-to-market strategy is that we got around that.
[00:16:23] Luke Freiler: So, but, but we lived with those, I mean, it funded our first year. I can never say it was a first mistake. It really did help put the business on the map, gave us an incredible customer and a wonderful relationship and all sorts of social proof. But it also kind of stuck us to our guns way longer than we should have because we didn’t accept that the context changed and.
[00:16:41] Luke Freiler: Now any company that’s grown up in the last 10 years or so, uh, is agile. They’re, they’re typically doing consistent development of some, you know, most products or services now. So our newer customers, the Pelotons and Sonos and Rokus of the world, they all have teams for this. They all centralize efforts for this, um, because they have a constant need.
[00:16:59] Luke Freiler: Therefore, again, it went back to that idea that, that we need a tool or, or services to support that. So yeah, it’s been an interesting journey that’s been in kind of in all directions. No, that’s really interesting.
[00:17:10] Matt Paige: It’s the, it’s interesting at that initial frame of reference, it’s just kind of burned into your memory almost.
[00:17:16] Matt Paige: Like you talked about, like, you were always working from that point and it, it was kind of, uh, you know, it taking you down the, the wrong path in a sense. That’s, that’s really interesting. Uh, let, lemme get back real quick to, you know, the product wasn’t fully ready in the beginning either, you know, and you mentioned y’all were bootstrapping this because you were coming out of effectively the.com.
[00:17:39] Matt Paige: Boom and bust. Right. So funding was, it’s not like it was by choice per se, it just wasn’t a,
[00:17:44] Luke Freiler: it didn’t even, didn’t even try to raise money. No, no point. Yeah. And I
[00:17:47] Matt Paige: think this is an underserved thing from a lot of companies is, you know, services is, is a lot of times a very viable way. A kind of help fund what you’re doing, but you also learn as you go.
[00:17:59] Matt Paige: I know with Hatch Works that we’re, we’ve been bootstrapped as well. We are a services company, so it allows for more of that. But talk me through that journey, because I think that’s a really interesting story, how you got kind of into the services game. Yeah. And it’s evolved over time,
[00:18:13] Luke Freiler: so, so yeah. Even before that big software deal, you know, we had spent about, I don’t know, nine months or a year kind of building up the platform and.
[00:18:20] Luke Freiler: This point we’re taking on separate contract work. ’cause we’re all like 20, 21 years old and, and don’t have life savings. So at, at that time through a friend of a friend, we met this, this brilliant guy named Mike Fine. And, and Mike had spent a bunch of years running beta teams at, at a few big tech companies.
[00:18:37] Luke Freiler: And at the time, he was the most passionate person I had ever met about this whole concept, this whole domain. And when he heard we wanted to start a business about it, he was just instantly all in it. Just, it was, was in his blood and what his perspective was. And, and the pitch he gave us that, that to this day is, is still alive was, Hey, while you’re building the software, why don’t we deliver managed services on top of it there.
[00:19:01] Luke Freiler: Is a need out there. Especially if you look back at that sort of episodic need of, of once every year or two per product manager and they have a product. He said there’s a need out there and if we’re delivering on the software, it doesn’t need to be done. Right. The, the last 10, 20% of the product takes 80% of the time when you’re, you’re polishing and getting it ready for users and going through the UX and everything else.
[00:19:20] Luke Freiler: So, you know, when we’re working close in hand within the company, we don’t need that level of polish. We can do the, the heavy lifting behind the scenes. Mm-hmm. And, and he made a compelling pitch and, and we started working with him virtually immediately. And, and we started delivering managed services. And, you know, initially it, it was challenging because we’re now effectively running two companies in one.
[00:19:39] Luke Freiler: We wanna be a software company. Mm-hmm. We also have managed services, but at the same time, those services lean on the software heavily. They’re a tech enabled service. And that means they have pretty great margins. And, and as we innovate the software, we can directly have an impact on the margin of our services, which is a really great thing.
[00:19:55] Luke Freiler: So to this day, we still offer managed services. It’s still a part of the business. And most importantly, as you’ve alluded to, it became what we think of as our test kitchen, right? We, we knew that we now had this central team that ran a more diverse set of, of beta tests than virtually any company in the world.
[00:20:12] Luke Freiler: And therefore we could learn and implement and. From that, we started to recognize that, hey, one of the problems in our space is that there’s no framework, that everybody does it differently. Everybody’s kind of recreating it from the ground up, and everybody’s wasting a lot of time, and the opportunity cost there for them is huge.
[00:20:27] Luke Freiler: So what if we stopped thinking about just technology and started thinking about the process and everything that’s layered on top of it that companies could reuse and b, That we are in the center of this and do so many of these tests. We don’t even have to go learn from everyone we can and we do, but we have it direct.
[00:20:42] Luke Freiler: Like we have the information that they would never wanna share with us anyway, right? It’s, it’s right in our wheelhouse. So, so that led us to making a framework and certifications and, and everything else, and just kind of mastering our space and recognizing that if this category is going to flourish and, and achieve its goals, um, it can’t just be software solving a problem.
[00:21:00] Luke Freiler: And, and therefore you need, you know, in, in it, the typical way is, you know, software process and people put those three sort of things together to achieve something. And we recognize that we had the software and that’s what we were focusing on. The people were our customers. And ironically, the process we’d already been doing, but never really thought about it as a gap.
[00:21:18] Luke Freiler: So, it’s been a while now since we implemented that. It’s been pretty successful for us. We’ve been very happy with it. But our goal was to just get rid of that first thousand hours that every company was gonna have to use to, to reinvent their own wheel. And it’s been, it’s been great. Mm-hmm. So services to this day are a big part of our business and, and I likely will be indefinitely.
[00:21:36] Luke Freiler: Y you kind of hit on the
[00:21:37] Matt Paige: topic there. It’s the, the dog fooding, uh, uh, um, element of it, right. Where you’re using your own product to enhance the product. H how do you think of that in terms of, you know, thinking about your own RO roadmap for your product, how you think about Sure. You know, prioritizing your backlog, like what’s going through your mind, because there’s all kinds of different levers from taking.
[00:21:57] Matt Paige: Taking care of technical debt to new features, to, like, what, what’s your approach to
[00:22:01] Luke Freiler: that? So, I, I’m uniquely positioned to, to think about that probably more than most CEOs because I am very much a product driven, C E o I, I still lead our roadmap. I’m still, I, I live in Figma, like a product manager would.
[00:22:13] Luke Freiler: It is like, I wake up Saturday mornings and, and work in Figma for fun. Uh, I’m just that kind of nerd. So for me, I, I’m very, very involved and I beg my accounts team to get me roadmap meetings with literally every customer possible. I show them everything we’re working on. I take feedback in, in real time.
[00:22:29] Luke Freiler: I prioritize it from there. Um, it, it, it is a process of, you know, we are on a, a very agile process where we have a release every week, uh, which is typically faced facing, bug fixes and small improvements. And then we have a feature release, a significant feature release every month. At any given time, we’re working on three to four months of those with different subsets of dev.
[00:22:49] Luke Freiler: So for us, it’s very interactive with our customers. We’re in. The process right now of shifting. In fact, we’ll be announcing it. It, it actually might already be on our What’s new page, but if not it will be this weekend we’re shifting to a feature flag based, uh, beta program. So in the past we’ve had what we call our Go early program, which our customers can opt in and they can either opt in their main implementation or they can opt in a stage.
[00:23:11] Luke Freiler: And we have a unique challenge in that beta testing. The beta testing platform is a snake eating its own tail, and we don’t. Don’t want to create friction through an unfinished product for their customers who are dealing with an unfinished product. Mm-hmm. It multiplies on itself. So what we’re doing now is, is the, the more what I would consider modern approach of being able to flag on and off certain features on demand for different customers, and basically pre-leasing everything at least a month ahead of time.
[00:23:35] Luke Freiler: Giving them some time to, to adopt it and give us feedback on it, implement those, those improvements and changes, and then roll it out in the next month and roll. Out the next new thing. So this month, actually, as of this weekend, we’re gonna announce what the first sort of feature flagged feature is.
[00:23:49] Luke Freiler: Everybody’s who the customer is able to kinda opt in and try it out. And then as of this month on what we’ll have those with, with each new release. So, but uh, you know, our main process is, is to dog food our own stuff, to use as many customers as we can. Uh, there’s a lot of challenges with that. Like I said, snake eating its own tail and our customers are very busy, but, Thankfully, we got a lot of tactics and tools to, to get past that.
[00:24:11] Luke Freiler: So it’s, it’s been pretty successful for us. And you
[00:24:14] Matt Paige: talked about Agile in this, uh, process of releasing every week and all of that. How long has that been going on? Like, is did y’all, were y’all agile from the very beginning, back when it was this new novel concept of something? No. No. So,
[00:24:27] Luke Freiler: so, so yes and no.
[00:24:28] Luke Freiler: In fact, I, and I think many companies have gone through some variant of this, and, and I’m always mm-hmm. Willing to, to follow my sword and talk about it. We were. Agile from a very early age, and, and I just fell in love with it. I, I think I was CTO at the time and I remember my first write up to the company about, you know, how interesting and, and amazing it would be to finish a feature and get it in in people’s hands, right?
[00:24:47] Luke Freiler: The main pitch for Agile was rather than doing a big release, you know, in frequently where half the features are done and sit on a shelf for six months while the other half get done, um, And therefore they’re not producing value for your customers. Mm-hmm. And when people are paying for something, you know, monthly or annually, they want persistent value, they’re paying for a service.
[00:25:05] Luke Freiler: Uh, it just clicked in my head, like, there’s a few things in my career that immediately just like, oh my God, that’s amazing. Why aren’t we doing that? Content marketing was one. I absolutely love it. This was another, um, so we tried it and, and we built our own system and, and we did it kind of our own way and, and we failed miserably.
[00:25:20] Luke Freiler: Uh, it just did not go well. And part of it was at. At that point, the enterprise was not super savvy to agile delivery or acceptance. In other words, they didn’t want releases to come that quickly. And when you have a company who builds a process and builds a team on top of a platform, they. Need, you know, a lot of, of, of time to redo kind of how that team works and anything you change can disrupt that team.
[00:25:48] Luke Freiler: So we, we were at this situation where we were capable of releasing monthly, but we have customers who are basically saying, yeah, I only want it quarterly, or I only want it every year even. And, and we’re sitting here going, okay, well we’re gonna roll whatcha talking about every week. These guys are gonna get it every month and these guys are gonna get it once a year and we’re gonna support oh man, 11 that they don’t have.
[00:26:07] Luke Freiler: And, and it was just a nightmare. So, Eventually, we basically walked away and said, okay, we’re not gonna do Agile anymore. We’re still gonna operate Agile internally, but we’re not gonna release Agile. Because we’d gotten used to it and started to like it, but our customers weren’t accepting of it. And so then at that point we’re releasing every like three to six months, we’ll do a big meaty release just because at that time our customers were pretty much exclusively enterprise and, and that’s what made sense.
[00:26:31] Luke Freiler: And as we built a bigger team and our engineers got more confident and we put project management instruction in, in place and whatnot. We started to get pretty confident that we could do the rollouts in a better way. And more importantly, the, you know, this is what happens. Everybody kind of matured and it became the norm.
[00:26:49] Luke Freiler: And it’s, you know, it’s the companies like Slack and whatnot that don’t give you a choice and therefore everybody kinda got used to it. So now, and I’d say for about a year now, no, two years now, um, We’ve been, yeah, more than that. I don’t know how long. For a long time now we’ve been pure agile doing these monthly releases, you’d have to go through the mega backlog.
[00:27:07] Luke Freiler: That is our, our What’s new site. Uh, and, and our customers seem fine with it. Every once in a while you’ll get a little bit of pushback on something, um, because we didn’t do a good job of, of prepping them for it. And we’ve done some interesting things that, you know, we, we question, we constantly experiment.
[00:27:21] Luke Freiler: Like we always release our release notes early now, and. I still question if that’s the right idea or not. There’s pros and cons to it, but at least if those customers are interested, they can get a sense of what’s coming. They can ask questions ahead of time because they might not want to be part of the test.
[00:27:35] Luke Freiler: They might just want it when it’s out, but still, you know. Yeah. Wanna know. So we’re constantly experimenting. It’s a challenge and it is directly proportional to sort of the size of the company if they’re a slow moving, very, very large company with a big infrastructure. Are, they’re not gonna adopt most of your new functionality anyway.
[00:27:50] Luke Freiler: So you have to find a way to roll it out that doesn’t impact them. And the smaller guys like want every bit of value and, and efficiency they can gain. So they’ll take it as quickly as you can give it to ’em. So we try to make the best of both worlds. Yeah. Yeah. Interesting journey there.
[00:28:04] Matt Paige: Uh, kind of, I got a back and forth a bit.
[00:28:06] Matt Paige: Uh, so I wanna get into, you know, the hot topic right now, whether it’s. Hype or not around generative ai. You know, the, the concept of ai, even large language models been around for a while, but the hype around it and the access to it is just accelerated. I’m curious from your perspective, how does that impact, you know, your business, your product, the concept of usability testing?
[00:28:30] Matt Paige: Or are there any just like, uh, you know, views you have of the future of how things just completely stepwise change from what they. Are today related to your world, or do you see it as this, you know, just it’s hype and it’ll die down and, oh, not
[00:28:45] Luke Freiler: much, much change. I, I, I definitely don’t see it as just hype.
[00:28:48] Luke Freiler: I mean, obviously there’s a hype component to it, but I, I think it’s incredible. And I, I’ve been mm-hmm. Fascinated since like day two, uh, my. My marketing VP was fascinated from day one, and he spent a few days trying to get me interested and, and I’d played with GPT when it was a little earlier on and it wasn’t as impressive.
[00:29:03] Luke Freiler: Mm-hmm. And then the thing that turned me was when, well, two things. One, he had to write a rap about our company. And what was amazing about that was he didn’t tell it anything about our company. It just knew, and, you know, we’re, we’re a pretty small player for a site we’ve never heard of at that point in time to know about us.
[00:29:19] Luke Freiler: But, but it wrote a, a very accurate rap about our company and culture that just broke. My brain. And then the next thing he had it do, which again to this day might be one of the most impressive things I’ve ever seen it do. This is like last November was he had it write a, a social post that was, um, Thanksgiving themed.
[00:29:37] Luke Freiler: It was holiday themed. And what was amazing about it is it created its own analogy. All, all he did was said, I, I want a social post that’s Thanksgiving theme for my company Centercode. And it knew that we were in the beta test management space and it made. Made this insane analogy about how beta testing is like the, the final spicing of your food.
[00:29:57] Luke Freiler: And if you don’t, you know, put the spicing on at the right time, oh, your, your meal’s gonna be a failure. And I’m sitting here going, I, I can’t process mentally how it came to that. We’ve been doing this for, for 20 years and we’ve never thought of that analogy. And there’s a temporal aspect. I mean, it, it really is incredible.
[00:30:14] Luke Freiler: Uh, It was very shortly after that I basically went to my team and I still subscribed to this line of thought of like, look guys, um, AI is not going to replace you. Someone who uses AI is so, let’s be that good. Let’s be that person. Right? And I, I kind of put it out there of like, let’s imagine that AI is, is saving each of us, you know, 20 minutes a day.
[00:30:34] Luke Freiler: And let’s imagine the company over there isn’t using at all. How much of an advantage do we have as a company? Once you sort of roll that up, that, that. We really have, you know, no choice not to say yes to. So I’m very positive on it in that regard. Um, at the same time, I have all sorts of reservations of just kind of how I mature.
[00:30:53] Luke Freiler: It’s right. One of the interesting things is there’s just no such thing as a mature AI product right now. Uh, we just did our entire performance review program process with an amazing tool. Um, I will give them a shout out. It’s called Hyper Context. We use it to. Manage meetings. Mm-hmm. Um, but they invited us to try out their new tool that uses AI to write performance reviews and kind of imagine a scenario where you are already collecting a year’s worth of, of meeting notes, um, for everybody in one system.
[00:31:21] Luke Freiler: What could you do with that in sort of understanding how people are performing and whatnot? It’s just a fascinating to think about, but as you. Might imagine as, it’s amazing, it’s still immature, so, you know, yeah. There, there is nothing that’s mature yet, so we can’t rely on it. They approached it very smartly to where it’s basically giving you a starting point, which is a great way to think of ai.
[00:31:38] Luke Freiler: They’re not trying to say, here’s, you know, an AI review, just hand it to them and walk away. They’re basically saying, here’s everything we learned, now edit it and, and make it your own and make sure it’s accurate and whatnot, which I think is the right way to approach ai. Um, we have a bot in our tool that.
[00:31:52] Luke Freiler: We started developing about three years ago, and there, there’s some, you know, re religion to this, but we refused ever to call it ai. We, we never thought of it as ai. We still don’t think of it as ai. Um, we certainly think of it as machine learning, kind of a tier below ai, but it’s not what chat g p t is and on.
[00:32:08] Luke Freiler: One hand you could say, oh, that, what I’m really saying is it’s not as impressive as ChatGPT is. And that’s actually not what I’m saying. What I’m saying is it’s not as, as powerful or unpredictable as ChatGPT is. Mm-hmm. And in our world, what our bot does is effectively manage all test or engagement for you.
[00:32:25] Luke Freiler: That the biggest problem in our space is, is getting people to engage and participate and give you feedback. And we built a bot that builds sort of a behavioral profile on every individual communicates. Them through the channels that they prefer, at the times that they prefer, simply by watching how they perform and what they do.
[00:32:41] Luke Freiler: And all of its messaging and everything is unique and it’s a bot. And our, our sort of go to market was, we’re not gonna say it’s a bot, but we’re also not gonna say it’s not a bot. We’re just gonna kind of see what happens. And most importantly, we can control the code 100%. We know exactly how it works, which is not true of most generative AI and sort of the black box of it.
[00:33:00] Luke Freiler: Um, so we. Know it’s not gonna go rogue and, and piss off somebody. Um, but we also, uh, know that it’s not sending data anywhere. It, it’s not, it’s all within our systems. It’s all controlled and whatnot, which is another big AI problem that most of our customers have. So we’ve ran contests internally and prototypes a million different ideas and whatnot.
[00:33:19] Luke Freiler: We’ve run contests to think of every single way that AI could benefit our company as a tool, every single way we could implement AI into our, our platform as a tool. And we’ve got. At this enormous backlog of incredible ideas, but we have to be careful with it. It, it has to be reliable, it has to be somewhat predictable.
[00:33:36] Luke Freiler: If people are gonna sort of put their brand behind it, they need to be sure that it’s not, you know, Skynet. So, so that, that’s a lot. But I, I’m in love with it. I think it’s fascinating. I was so excited, like in the early days of showing my mom and my, you know, my kids, uh, it’s been just mm-hmm. Fun to watch people’s brains explode and, and try to figure it out.
[00:33:57] Luke Freiler: It’s, to me, You know, it basically went the pc, the internet, the mobile phone, and this. Those are the, you know, major technological breakthroughs we, we’ve had in our lives. And this one is just as big as the rest of my mind and, and could be bigger. Yeah. I’m with you
[00:34:13] Matt Paige: there. I think it’s, uh, on that same level, let me ask you this.
[00:34:16] Matt Paige: Do you see a future where you actually have a generative AI agent or, you know, bot, whatever we want to call it, doing the using. Ability testing instead of an actual person. Is there ever, is there some, you know, utopian future where that’s the case, or do you think it’s always you need that human involved, uh, in
[00:34:39] Luke Freiler: the process?
[00:34:40] Luke Freiler: I, I don’t, I would not be comfortable at this point in time predicting too far out on anything with ai. Mm-hmm. Um, that’s safe. I, I just, I, I don’t, yeah. I, I, I, I’m not, I’m, I’m looking to hedge my betts, I guess. But, you know, I, I, again, I would’ve never imagined it could’ve written a wrap that was weirdly sensible about my company without me telling anything about my company.
[00:35:02] Luke Freiler: Is, is just unwinding that is, is still mm-hmm. Uh, fun to think about and endless. I’ve never felt both more excited and overwhelmed about a technology in my life, ever. It’s just nothing’s happened this quickly. So the rate which it’s happened is impressive. But at the same time, I do think we’re about to see a lot of diminishing returns.
[00:35:22] Luke Freiler: I, I think they kind of had it under wraps for a long time while they did a lot of the back work and, and then they exposed it and it moved really quick for a little while. But again, I, that’s ’cause that was already done. They just hadn’t really rolled it out yet. They weren’t confident in it. Mm-hmm. Now I think we’re starting to see a little bit of a slowdown, but now, What’s cool about AI is we’re moving from the tech being the impressive part.
[00:35:42] Luke Freiler: And now the impressive part is the creativity of all of the developers who now have access to that new thing and can think about entirely new ideas that, that aren’t out there. Mm-hmm. So back to your question though, I think there will be a level of, of user testing and product testing that happens in the AI space.
[00:36:01] Luke Freiler: Uh, I mean, I, I know there’s already companies working on it. We’ve talked about it, we’ve thought about it. It’ll happen. Will it ever fully replace? I doubt it. Um, in part, again, not knowing where it goes, who knows if it’s, you know, on a robot in your house, they can walk around, then it’s a whole different game.
[00:36:16] Luke Freiler: But, um, there’s a lot of things that, that happen in the physical space and have to do with humans and whatnot that are very, very, very difficult to predict. But could it, you know, like all things take out the, the 20% of stuff that wasn’t quite obvious but is predictable. Yeah, it, it could definitely do that and, and will, but depending.
[00:36:35] Luke Freiler: On the product. Just the nature of, of what technology is. I think you’re always gonna have to be testing what comes next. And I don’t think, and anyone
[00:36:45] Luke Freiler: and
[00:36:45] Matt Paige: anyone looking to go a little deeper on the generative AI side, check out our previous episode with Jason Slacker. Um, you mentioned the idea of like, you know, if everybody could be 20 minutes more productive, he kind of framed it as, imagine you have a thousand interns or a small country trying to get people into thinking, framing out.
[00:37:03] Matt Paige: Side of their, their normal, uh, working model, which is kind of an interesting or mental map, I guess I should say. Interesting way of kind of framing things. That’s
[00:37:12] Luke Freiler: Yeah. That’s, that’s very savvy. Yeah. But
[00:37:16] Matt Paige: Luke, uh, really appreciate the, the time today. Uh, and where can folks find you, find Center code? What’s the best way to find that?
[00:37:22] Matt Paige: Yeah, a absolutely.
[00:37:23] Luke Freiler: So center code.com uh, is is the website, as you might imagine. Um, my email is just firstname.lastname@example.org. I’m, I’m not nearly as active on, on the social media as I should be. So while I am. On LinkedIn and, and you can connect with me and whatnot. Uh, I’m also just kind of an old school emailing guy, so feel free to, to shoot me a message.
[00:37:40] Luke Freiler: Um, you know, our product is, is very different than the stories we told. You know, initially it was aimed for big teams and whatnot, and in recent years what we’ve done is evolve it to really target product managers and, you know, all the way starting at, at free to run a whole program on your own to a very inexpensive self-serve.
[00:37:56] Luke Freiler: And, and within just a few minutes, you’re gonna be recruiting your own customers. And, you know, I think it starts. Said about 40 bucks a month. It’s really, really simple and straightforward. But then as your program matures, if you do choose to centralize and whatnot, the platform can grow with you to more of its traditional base.
[00:38:09] Luke Freiler: So, um, kind of options for everybody. If you’re looking to engage with your audience or even do internal dog fooding, anything that involves using, using your product and gathering, you know, feedback again, issues, ideas, praise, or whatever else is gonna help improve your product. Um, check us out. You can try it for free, you can play with it.
[00:38:24] Luke Freiler: And my team is, is built a product managers who are there to help you and, and talk about it with you. So happy to help.
[00:38:32] Matt Paige: Yeah. And I know we didn’t get into that, but that’s an interesting piece from the evolution of the 300 K to now this p l g motion where you can try it for free. And that was the unique thing too.
[00:38:41] Matt Paige: Like, I know we kind of just ended, but keep going. I it the, the fact that you have product managers almost acting as like your salespeople just kind of blew my mind a little bit. You actually have like knowledgeable people. People talking to the customers that actually know the customers in a
[00:38:55] Luke Freiler: sense, yeah, I, when I, when I talked about doing that and, and talked about why I wanted to do that, I, a sort of group of peer CEOs thought I was crazy at first.
[00:39:04] Luke Freiler: And, and I, I, as I’ve gone through it, I’ve, I’ve really enjoyed it. There, there is certainly a place for, for sales and for me it’s more about relationships with the customers and growths and making sure their needs are met. But what I don’t want is somebody on the phone with a customer who doesn’t have.
[00:39:20] Luke Freiler: The agency or the right motivation. Mm-hmm. You know, my dream org from a sales perspective, and, and we’re there now is, is nobody selling the product, has a commission. And, and that’s because I, I want them to be motivated to sell the right thing to the customer as opposed to the right thing to hit their quota.
[00:39:35] Luke Freiler: That that quota is valuable to the company. It’s how we we eat, but you sell the right thing to the customer, they’re gonna use it, they’re gonna grow, and it’s, it’s not gonna be an issue. So, um, yeah, I’m very proud that, that we have a, a product focused team and yeah, we have. Have the p l G method in to get people going and, and then if you do want to move up from there, we have experienced product managers who are literally the product managers of our platform that are stepping in to help with those conversations.
[00:39:58] Luke Freiler: And, and I’m really excited about that and I’m trying to use as much of my time to have those conversations as I can. ’cause they’re, they are, you know, one of the fun parts of the job, so it’s a good time.
[00:40:08] Matt Paige: That’s awesome, Luke. Well thanks for joining us today. Have a good one.
[00:40:11] Luke Freiler: Thank you, sir.