So did anyone have any realizations about the metrics that they're currently using on their company? Yes. Can you share a little bit. >> All right. Hi everyone my name is Nicole Winston, I'm from Verizon. I would definitely say that my metric is actually financial, all about revenue and then the revenue goes back and gives me more funding for the next round. Engage my customers. But just to give you a little background, so I am our product manager for our software management services for IoT. And that means anytime a device needs to be updated in the area, we'll go to our product to actually go ahead and execute those firmware campaigns. Now from a customer perspective. I still want to check my devices. So I need to make sure that I look at latency in the time to take them down in the success rate, which are not our executive metrics. Or executive metrics always come back to revenue. Even though those are technically things that I'm concerned about they're not in leadership. >> Yes. >> Do you think you guys can maybe start introducing those to executive team? >> Yes, I think it has to be strategic in the way we introduce them. >> [LAUGH] >> Anyone else, did anyone else have any realizations? About the metrics that they currently use or ways to improve them. >> From my Salesforce friends here, I'm Meredith Brown from Salesforce. And one of the things we measure a lot of monthly active users monthly active or just to see how many customers are using it. But one of the things that we like to talk about on the product is we help them do certain things, like grow their business, find leads. Wouldn't it be amazing if we actually measured that our product helps them do those things? >> Yeah. >> We think of customer stories, but if we could do it in a more operational way, that would be pretty amazing. >> Hi... ...Irina Farooq from Kinetica... I was 13 years ago and she worked at a startup for CEO who came out of sales. But his guiding principle was customer satisfaction. And one of the top measurements for himself and for the entire company was customer satisfaction. And then every quarter he had every employee present their goals and objectives in front of everybody else. And everybody had to come up with a way they could improve customer satisfaction. And it was literally the most beautiful thing I have ever seen like in the environment of club because the engineers are like documentation people, people would come up with all these ways to improve or address customer satisfaction. And engineers think, "I can refactor this API to make it more efficient or I can respect to that mutation" and all these things. And you could see and then having actually measuring customer satisfaction. Have you set a thing that you measure in a survey and you follow up. And it was really, really great to see because as a product person, it's a set of things that I feel passionate about in front of me. But I don't have full visibility into the death by 1000 cuts, the whole interaction of our company as a whole not just the product and the customer is experiencing the Verizon. They don't just experience with your IoT product. They experience Customer Care Center and, a store and everybody else and it was important to have that integral. So, I'm working my hardest to was wherever I am, come as close as I can to that metric for injecting the right and even now go into the planning process and I go from engineers bottom up to make sure what is the technical debt you want to pay? What are the things that keep you up at night? To just find out what are the blind spots. What are the things that will hurt the customers going forward that are outside the news cycle. >> One more? >> Sure. Testing. Hi everyone, I'm Arianna. I'm a product manager on Google on Google Search and actually tying together two things here. And then I want to tie it back to something you mentioned earlier and then end with a question, which is so the team that I'm on right now, we are in a period of transition moving from pretty like high level genericized metrics like DAU and MAU, which on a product, as mature search can be pretty hard to move and actually measure to thinking about customer satisfaction, and then things like task completion, and that's moving from like, just a metric and a dashboard to actually the way we thought about doing those through surveys. And I'm wondering though, back to your question your comment earlier about like surveys and tests like not your actual customers. Is the distinction there that what you're suggesting with the surveys and tests, it's not a good way to cultivate new ideas and new directions as opposed to surveys are totally okay from a measuring success and outcomes perspective, that's my question. >> Yeah. >> So the insight there was really that quantitative data and qualitative data needs to be married together, right? And that's why I bring up that quote when the anecdotes and the data conflict trust the anecdotes right, because you need to have that why that intuition that are we, so we can be measuring something, but it might not mean what we think it means. So we need to really have the two work together. And yes for new product development for intuiting, where the world will go where our customers will be 12 months from now or 18 months from now. No survey will get you that right because customers won't be able to answer that. So for that you really need to develop that deep intuition. But, for things that have been in market for a while, yes, you absolutely need to generalize your understanding about what's working. So yeah, all right. So finally, just the three things that I hope you will bring back to your teams is, that to use the customers the Northstar, to try to resist proxy's for customers. To tell your vision and your values story through your metrics. Right, so your metrics should reflect your vision and values, they should not be separate things. And to prototype with customers early and often. I hope this will lead you through some very strange and tumultuous times in product development, thank you so much. >> [APPLAUSE] >> Thank you for helping us grow. By the way, Tatyana is also our executive advisor, so Thank you for helping us grow AWIP! [MUSIC].
