Episode Transcript
[00:00:01] Speaker A: Welcome to Pixel Parenting, the podcast that helps families raise kids in a world full of screens using science and curiosity. Here we share science backed strategies and real conversations to help you build healthier digital habits with practical tools for everyday parenting in this digital age. Let's get into it.
I'm excited to welcome Vivian Chung to today's episode. Vivian is a co founder and CEO of upguard, a youth AI safety startup that helps companies build AI tools that are safe and healthy for kids. Her team creates evaluation and monitoring systems that catch things like self harm and sexual explicit content in AI Companions and ed tech tools. Vivian brings years of experience in trust and safety to one of the most urgent questions parents have right now. How do we make sure AI is actually supporting our kids well being and not quietly harming it? In this episode we'll talk about what safeguards and guardrails really mean inside AI products, how AI companions can both help and hurt kids mental health and what's happening with the new California AI bills and most importantly, concrete things parents can do to test and evaluate AI tools at home. Thank you Vivian for being here. I'm very excited about this conversation from a real startup making their way into the AI world and here, here in Silicon Valley. Could you please introduce yourself first and then we jump into how you got here and what AppGuard is about.
[00:01:29] Speaker B: I'm very excited to be here as well. Hello everyone, I'm Vivian.
I am co founder and CEO of appguard AI. We're a youth AI safety and well being startup. We are helping AI companies build safer and healthier products for kids and building real tools to help them embed that within their system architecture. Both me and my co founder come from trust and safety experience.
We both have a real big passion for making spaces online spaces healthier and safer for everyone.
And we knew that we wanted to bring that experience to the world of AI.
And earlier this year we're selected for a grant that we applied to. It was affiliated with UNICEF and more focused on online child sexual exploitation and we had pitched them a tool, an AI evaluation tool to evaluate for online child sexual exploitation risks. And we knew we wanted to build a company as well and we also wanted to build in the public interest. So the grant was a really great way for us to start apguard off on a public interest footing. But we also knew that there was a big opportunity to build a startup and create more accessible tools for the broader AI industry to embed within their products. We're still working on the AI evaluation work and we're working with civil society partners, industry partners to really build that out. But we've also been just following the news and media very closely and seeing what market demands there are, which is why we've expanded to also include youth well being in our AI evaluations. And also the tooling that we offer to support for AI companies and their efforts to make their products healthier for their kid users mental states.
[00:03:33] Speaker A: How does upguard exactly work with the other company's products? How do you make sure that it is safer for kids?
[00:03:42] Speaker B: We work directly with companies, for example, AI companion companies, companies building like products specifically for kids. Everything has AI in it nowadays, right? And the range of what kids use in technology is always evolving. So I don't want to limit us to say like oh, we're only working with EdTech for example, but that that is included. And so we, we work with these companies and we just better understand like how are they designing their product for kids particularly and just understand what their concerns are and building for youth. I think this is our target audience for now. We can talk more about how every company is building for kids nowadays later on. But right now our target is those organizations and companies that are specifically targeting youth. And we just better understand what experts are they engaging, how are they evaluating the AI and understanding the AI outputs that are being generated from their models? And we work with them to help them better understand what their AI models are capable of. And we then also plug in our tool which one of the tools we're building for example, is doing some real time detection and classification of what types of content is being ingested in these models and help build reports and help these technical teams build a better understanding of the types of content that they're receiving from users so then that they can better respond more appropriately to the user in real time. And so we're building an automated pipeline for both the user to have a better user experience first of all and be routed to either better support or a better choice of responses. And then the technical team, they're able to just better understand how their users are actually engaging with their product. And so we're starting off with kids safety, but it's truly just supporting the broader user experience as well.
I'll just use an example of a mental reflection tool used by schools, for example.
The intent of this product is to support the kids well being, but the tool might be using AI to help engage with the kid or providing feedback on the kid's progress over time, for example. There's a lot of great opportunities with AI to help enhance the kid's experience.
But obviously with those usages of AI, there's some risks. So for example, a kid types into a text box.
It could be AI chatbot, it could even just be a standard text box, like, oh, I'm feeling, feeling sad today, I don't want to do my homework. We would detect that as something that doesn't really need any sort of escalation. For example, to a teacher or a mental health authority. It's just like a kid is sad. A kid is supposed to be able to express themselves with the technology tools that they're using. We don't want to stop that. But then there could also be the flip side of a kid using this mental reflection tool and really just like, like using it like a real life journal.
Oh, I don't want to do this anymore. For example, I don't want to go to school anymore. And some other like, harmful or phrases that indicate towards a more severe case of mental health distress.
And so we would help detect and flag the responses that the kids are using. And if, like, there needs to be some sort of escalation to a trusted adult or whatnot, we would help support that workflow. So it's both like the detection and appropriate response. Right now, the default solutions on the market, they either like, refuse outright and don't even let the child engage, which might then lead to a child going into more distress and not feeling like they have an outlet to express themselves. And then they don't get the help that they need. Or the model defaults to a pretty harsh response. Maybe like the kid wasn't really saying something that severe, but the model is flagging. Like, you need to see crisis support. Here's a hotline.
Instead of truly trying to engage and understand what the kid is trying to say. So I think there's a lot of opportunities, especially with tools to help kids express themselves.
But there needs to be safeguards and tools to help understand and route these conversations appropriately.
[00:08:51] Speaker A: You mentioned safeguards. What are those?
[00:08:55] Speaker B: I would define safeguards as any tools that are embedded within a technology product that help identify, understand and respond to risk.
And in this example of the kid showing some signs of mental distress, safeguards would be like, okay, is there detection with the product to understand and catch the signs of distress?
How good is that detection? So then it can route appropriately to the right response mechanism. What are the response mechanisms like? I would say safeguards are any like, layers of safety on top of the actual product itself. Because this product, example product we're talking about is just A mental reflection tool. If like the school provided this to a kid out of the box with no safeguards and there is some AI embedded within that, we know with Gen AI it can go off the rails pretty easily.
[00:09:59] Speaker A: What would be going on the rails?
[00:10:00] Speaker B: For instance, it could help reinforce some of these distressing behaviors like oh, I don't want to do this anymore.
And then the AI feedback tool would be like, oh yes, you shouldn't take care of your mental health. Sometimes that might be appropriate. Right? But it could go to towards extre we see in cases of the teen suicide cases of AI companions, it could really go to the extreme like oh, here are some methods for how to harm yourself and unalive yourself. The defaults in these models tend to be sycophantic. The models will try to just appease the user and give them what they want. But if a child or any user is not in their right mental state, what they want might not be what's best for them.
[00:10:48] Speaker A: So you mentioned AI companions. How would you define an AI companion? Because you just mentioned a tool that you know for mental health. Does that fall within the AI companion category or not?
[00:10:59] Speaker B: Definition of AI companion is pretty broad. Honestly. It's just like any AI tool that anyone can have an open ended conversation with and form a parasocial relationship with.
I mean the most obvious forms of AI companions is like a character AI where there's a character affiliated with this AI tool. So then you actually think it becomes your friend. Right. AI companions are a lot more insidious and not as overtly companion like even like general purpose chatbots. It's just a text box, but it gives you such flowery language and tells you you're right all the time.
And then like you can create a parasocial relationship with that.
[00:11:43] Speaker A: And does Upgaard look at that too? Like how it answers? Is it trying to create empathy with the user? Is it trying to be nice? Is it dry in the language? Do you look at that too?
[00:11:56] Speaker B: We've already been evaluating outputs based on like tone for edtech tools based on like how verbose and age appropriate it is. Even just in length. Right. There's many ways that we're already using statistical measurement tools to assess the outputs of AI. But we want to go a step further than that and also measure AI outputs and make sure that AI outputs that are coming out are aligned with behavioral therapy best practices are aligned with developmental psychology best practices, especially when it comes to the context of children.
Their brains are still developing, of course. And a general purpose chatbot for example, is like, designed for the default type of user, which is an adult.
[00:12:46] Speaker A: Is it healthy that AI companions empathize with the user and create that social relationship? Or would it be better for the kid's mental health to have a, let's say, drier friend or not as sympathetic? What is the right balance? Because you want to, I guess, and correct me if I'm wrong, what I'm thinking is you want them to engage, to learn, let's say for educational or mental health, but you don't want them to have a relationship with the AI. How do you think about that?
[00:13:20] Speaker B: My canned answer would be, it really depends on the use case.
But just generally this would be a good conversation to have between the technical team and any like, clinicians on staff or clinical experts that they have in their network.
And just better understand, like, what are like the most extreme risks that can happen with kids using this tool and then like work backwards words from there. I think it is okay for chatbots to express some sort of empathy. Kids are looking for that.
However, what crosses the line is when a chatbot is claiming that it feels that too. Or it's like claiming a human character trait that is beyond the scope of what a technology can be doing. However, it is like, just technically speaking, it is really difficult to patch all those edge cases, which is why in evaluation you try to do that up front and also try to tune the models so. So that they don't go that far. But of course, like, you never know what a kid is capable of, especially because they're so smart and they're so creative.
[00:14:30] Speaker A: Let's talk about the latest California AI bills that come out. What do parents need to know about those?
[00:14:37] Speaker B: California AI bills are definitely like, they've been making a lot of splashy news in the headlines. But there's other states like New York, Illinois that are also releasing a lot of AI bills themselves. And there's state patchwork of AI pills that's going on. But specifically with California, we have been particularly focused on SB243, especially because of its relevancy to AI chatbots and usage of AI chatbots. Others might be more around transparency from model providers themselves as to what they're doing. Whereas this 243 is way more like product focus and product aligned with what we're doing.
It's more speaking about AI content safety, for example, and what essentially says especially relevant to kids is these chatbots should not be producing sexually explicit content for under 18s and for all users, there should be like Self harm and suicide protocols if a user expresses that type of ideation.
And in the following year, companies will need to report out what those protocols are. So there's two forms of content detection and mitigation here. It's both who are sexually explicit for minors, and then also self harm and suicide content for all ages.
[00:16:12] Speaker A: How could parents check if an AI tool has guardrails in place? Is there a test that prompts that? They could ask that, you know, would make it obvious that it's well thought or not or safe or not.
[00:16:26] Speaker B: I would say on the parent front, unfortunately, it is a wild west and that's why podcasts like yours, resources like yours, exist.
I think every parent does need to have their own digital literacy strategy for their kid. It might look different depending on your own parenting style, but it could just start with a conversation with the kid about why do they want to use AI tool, what's their goal? And then having a conversation with them there. What do you think the risks of this could be? Just having that first conversation and then maybe the parent looking at the safety features, terms and conditions and if the parent can think of prompts that affiliated with the biggest risk that they both identified and just test to see how the chatbot responds or model responds, that could be a first step. I don't want to say like, oh, you should completely avoid all AI products.
As a child and a parent, how.
[00:17:24] Speaker A: Do you see the industry and what gives you hope about where it's going?
[00:17:28] Speaker B: I have a lot of hope for the industry. I know it's easy to be all doom and gloom seeing the media headlines of the tragic teen suicides, for example, but I mean, this technology isn't slowing down.
But fortunately the safety initiatives and the unifying initiatives across civil society are ramping up. There has been so much coordination amongst all parties across industries to try to get this AI think right.
AI is a galvanizing force to bring together people who otherwise would not be getting together because it is scary. But I think you need some catalyst to bring community together so I think we can overcome a lot of these challenges. I think there just needs to be more awareness amongst society as well.
[00:18:27] Speaker A: But I'm optimistic, I share that optimism. You know, like you're actually impacting the products, which is what directly impacts the children.
[00:18:35] Speaker B: Yeah, I think laws are important, of course. However, I do think there more can be done on the product side to just make it easier for companies to embed safety. I don't think any founder aims to start out an AI company thinking like, I want to do all these harms. Like, they, they just have this mission to advance the service that they have envisioned. Right. And that's why they need folks like us to kind of be their safety partner and journey with them as they build that service.
[00:19:06] Speaker A: Exactly. It's a teamwork. Yeah. Well, this is a great way to end it. Thank you so much. This was super interesting. Thank you for being here.
[00:19:14] Speaker B: Of course. Thank you, Patricia. Thank you for all that you're doing to educate parents about what's going on and making them feel like they have a stake in game as well, like they're not left behind. There's a lot that can be done and that is being done to involve parents in this conversation.
[00:19:30] Speaker A: Yeah, yeah. And they're part of the team. Right. There's, you know, guardrails, there's product and there's education and parents are part of the educational equation. So. So yeah. Well, thank you so much, Sam.
