Check-in: An AI accessibility feature to force users to take a break

Being part of the software design industry, I know how manipulative digital products can be if designed with business goals over user needs. If designed carelessly, poorly, or unethically, that hyper-focus on quantitative engagement metrics the software business can use to measure how useful and valuable a product is to people.

In business use cases, AI tools are unquestionably helpful and can expedite workloads, but some have become incredibly controversial based on how they’re designed and what they are used for. One concern is some finding them so helpful in different contexts, they are engineered to encourage people to use it more… and in some instances, more than is good for some users’ health.

Why is this needed?

We’ve already seen the harmful effects some digital products designed specifically to get people to use it more can have on at-risk groups.

As a designer myself, I started to wonder what the industry can do to help people recognize this more clearly and responsibly, especially who may need help to get an awareness of how manipulative and exploitive a consumer digital product can be, when designed by Careless People.

What to do about it?

I wondered what a more ethical AI product would look like for people who may easily lose track of time, especially those who are neurodivergent, whether they know it or not?

How could we encourage more intentional use of design patterns that aren’t solely created to drive “engagement metrics.” Who designs products, especially minors, with awareness of concerns parents have about technology they don’t feel like they have control over?

Then I remembered this:

Image of the Nintendo Wii Sports game screen recommending the user take a break from their gaming session.

This was a choice by Nintendo designers for Wii Sports. They knew how fun and addicting games can be, and were thoughtful enough to limit screen time before modern, always online platforms gave us modern parental controls over each platform.

Knowing we have a mental health crisis in the U.S., I wondered what this check-in style of messaging might look like for an AI product, especially in the consumer space. A more ethically designed AI tool with features that allow the user to ensure they take breaks would be an ethical design feature.

Tristan Harris explores in his design ethics consulting and helps offset what Cory Doctorow has been observing and calling the enshittification of products and services lately.

I thought it would be interesting to see if generative AI can be used for designing features to help people with this challenge.

In order to demonstrate what work and thought was put into this effort, I am sharing the instructions I wrote and entered as input into a few AI tools to explore how their outputs vary when providing design options.

The instructions below are written in Markdown, a shorthand of HTML, the markup language used to tell web browsers how to structure web page content, developed by John Gruber.

AI tools using large language models are designed to understand human-readable context, so the models generative AI is trained on can easily read this format and understand the context of the instructions you give it.

Download the Markdown instructions.

The Results

Mocking up a feature that accounts for usability and accessibility requirements, focusing on a feature that may help some who may be more vulnerable to tactics that take advantage of kids and those with neurodivergence, especially if they are not aware of their condition.

Helping an at-risk category of people who may be more prone to exploitation of AI products, especially ones designed to fabricate flattery and use sycophantic traits to drive up AI product engagement metrics.

A screenshot of Gemini’s solution

This is a solution from Google’s Gemini.

After some refinement, an in-line solution feels less obstructive and more suggestive.

A screenshot of Claude’s solution

Similar design and user interface patterns are used on dashboard systems on modern cars that appear used when the driver is demonstrating behavior that suggests they are inattentive, making them a danger to themselves and others.

A screenshot of the Kia Driver Attention Warning notification system

Replit designed this and generated an interesting brand name: Ethica, a great name for an AI product with ethical product design features in mind.

It took my product requirements with clear accessibility needs and delivered several user experience options that empower people to customize the product for accessibility needs.

A system of options can include check in features that ask the user how they’re feeling after constant use during long sessions.

An image and screenshot of the design showing a timer to inform the user the amount of time they’ve spent using the product.

The tone of the message is important. These can range in style and tone, from a slight nudge to obstructive. Scaling the restrictions for use of the product from a less libertarian option and ramps up acting more parental and cautionary, literally blocking the user when the user may want it to, especially if they are showing at-risk behaviors.

Image showing the system creating an overlay that blocks the user and suggests they suggest a break

These could be controlled with accessibility settings as part of an on-boarding user experience offering the user options like many other tech products do when users get a new device.

Screenshot of user-controlled settings that enable accessibility features

If the user demonstrates behavior that may suggest a temporary or permanent state of being in distress, mental health support could be offered.

Figma Make is an AI agent to help designers deliver product ideas.

This interactive prototype offered a heavy-handed approach showing all the options at once, which wouldn’t work and needs refinement. I would have to explore that further once I pay up to get past the paywall for more AI credits.

With more time, this could be an interesting system to consider to empower users and help them recognize when AI sessions go long.

At this point, I need a break.