Last week I was talking to a journalist about the 4-day week, and we got on the subject of flexibility. Of course, flexible work is all the rage now, and lots of workers (and some politicians or policymakers) want to see it become a mandatory option for everyone.

I’ve had reservations about this, and I finally figured out why.

For one thing, there’s a great literature on the real-world downsides of flexible work policies. Getting control over your own schedule benefits men more than women. The “flexibility stigma” slows down women’s careers. Men and women are rewarded differently for using flexible programs (or crafting the appearance of overwork). A study of work-home interference found that flexible schedules raised “expectation that you’ll do a great job juggling family and work,” and that the person with the flexible schedule becomes the default parent who’s expected to pick up kids or fill the gaps when something unexpected happens. While flexibility has logistical benefits, routines are also really good for us. And of course, you can make various arguments about how flexible work creates additional labor for everyone: just keeping track of everyone’s schedules, figuring out how to properly route information, etc., becomes harder than it is when everyone is in the office on more predictable schedules.

This suggests to me that even in well-meaning companies, flexible work is hard to do right, and that alternatives like a 4-day week seem to do a better job of rebalancing the needs of people, families, and employers. (Though I should note that the study of work-home interference also noted that work-family interference increased when people shorted their working hours.) There’s no suspicion that the person leaving work the early afternoon is creating extra work for the rest of the team when everyone leaves work early: turning this into policy creates incentives for everyone to cooperate around the goal of having a shorter workweek, rather than competing around the goal of having a shorter workweek.

But in the course of this conversation I put my finger on something else that bothers me about our discussions of flexibility.

When we talk about flexibility, we talk about it as if every day can be a make-your-own-adventure. The problem is that flexibility becomes a resource that the rest of the world can use to solve its scheduling problems. Rather than being something that makes your life better, it becomes a buffer that the world can count on when schedules crash, when someone gets stuck in a meeting and can’t do carpool, or a client demands last-minute changes in a deliverable.

Flexibility sounds like it should good for you, but it’s really another way for your employers to offload work onto you. Flexibility is a lot like mobility. We imagined that carrying our offices around in our pockets would let us break work into multiple sessions that would fit our lives, but in today’s 24/7 always-on world, it’s instead ground work into a liquid that seeps into our entire day.

So what’s the solution?

I think we should think of flexibility as a right, not a daily practice.

“Flexible work” should be like freedom of movement, or the right to marry who we want: it’s an ability that we can exercise in order to design and live more satisfactory lives. But just as freedom of movement doesn’t translate into us moving into new houses every day, and the freedom to marry who we want is something we ideally exercise once, flexibility should not be a license for everyone else to behave like chaos monkeys, or something that you should be expected to use to compensate for your manager’s inability to make resilient plans or your client’s inability to make up their mind.

So yes, let’s go for flexibility. But let’s think about how to use it to make our lives better, rather than just… more flexible.