Astro Hacker News - Show HN: Claude-thermos – keeps your Claude session warm for you

unholiness |next [-]

I've directly inspected calls for pro/Max plans and as of today they have 1hr cache expiries. This has definitely degraded to 5 min in the past but that's the behavior today.

If you're paying API rates, you can choose 5m or 1hr yourself (and pay different rates).

Keeping a 1hr cache warm could still be useful, sure, but outside that, I don't see much use of this today.

s0ck_r4w |root |parent [-]

When using API keys the default is still 5 minutes. Setting 1 hour for the entire session is actually a lot more wasteful considering the 2x rate that it comes with

jonas21 |next |previous [-]

Is the 5-minute expiration correct? I thought it was more like ~1 hour.

s0ck_r4w |root |parent [-]

You can set the retention globally (whole session) to 1 hour which will actually make it more expensive. The default is 5 minutes.

*UPD:* actually it appears the default is authentication-dependent. API key gets 5 minutes, subscriptions - 1 hour.

alukin |next |previous [-]

Feels like will be shut down real quick

gogobio |root |parent |next [-]

Yes, this wastes cycle. They dump your cache and deallocate the VMs so that others can use it. This will result in tokens wasted and downtime for others.

supern0va |root |parent [-]

Can you expand on this? I can't imagine this is holding up an entire machine or materially impacting capacity. Isn't it common to have a tiered cache and to ultimately evict (or move to another tier) if there's an active inference request and no available capacity elsewhere?

Lalabadie |root |parent [-]

Yes, however if a tool like this commoditizes a way to stay "higher" in the cache hierarchy to avoid getting dropped, then for the same total capacity, Anthropic have to become stricter about retention.

l1n |root |parent |previous [-]

Why? Seems fine if people want to burn more cache reads.

|root |parent |next [-]

agluszak |root |parent |previous [-]

Because it makes Anthropic earn less money

munk-a |root |parent |next [-]

> Because it makes Anthropic burn more money

Fixed that for you.

cadamsdotcom |root |parent |previous [-]

"Earn"?

Cache duration is arbitrary. What it actually does (if used en masse) is decrease the amount of oversubscription their infra can handle..

janderson215 |root |parent [-]

Earn is synonymous with profit, and this hurts their profits. Introducing latency can benefit much more people and actually make usage more efficient. Reducing latency can sieze up a functioning system.

Lately, I’ve been thinking about how this related to fractional banking. If you were to eliminate fractional banking introduced in the US by Hamilton, you would destroy a lot of current prosperity.

cosmotic |next |previous [-]

How will this not lead to tragedy of the commons?

davesque |root |parent [-]

Exactly. I really wish people wouldn't use this. If this becomes popular, anthropic will just modify their cache policy to be much less fair. It's not like they have infinite cache.

ATMLOTTOBEER |next |previous [-]

Glad this exists. It will force anthropic to fix their flawed cache mechanism.

cortesoft |root |parent [-]

How should they fix it?

addaon |root |parent |next [-]

There are a lot of workflows where cache is likely to be consumed exactly once (e.g. anything chat-like where a single thread is extended; if a prefix cache hit is found, and a set of new longer prefixes is inserted in the cache, it is unlikely that the original shorter prefix will be consumed in later turns). I could see allowing each session key a small number (1?) of 24-hour cache entries, where inserting a new cache entry (perhaps the maximum-length prefix, perhaps an API-tagged specified-length prefix) consumes that slot and either boots the rest of the items from cache, or demotes them to 5 minutes or something. Basically: workflow awareness, not uniform handling.

s0ck_r4w |root |parent |previous [-]

Make the cache TTL more adaptive. Have more tiers than just 5m and 1h. Long-running subagents become increasingly ubiquitous. There's no reason for Anthropic to not do a better job for this use case.

razodactyl |next |previous [-]

Don't we pay for cache input though?

broodbucket |root |parent |next [-]

You can send and receive 1 byte and refresh the cache. Keeping the cache alive isn't free, but it's close.

cortesoft |root |parent |previous [-]

The idea is this keeps it in cache so you don’t have to pay to re-input.

j45 |next |previous [-]

This is a good idea if it could manage it in an acceptable way.

For example, there might be something I intended to complete in one sitting, but took two sittings in the same day unexpectedly. Maybe it could just be a few cache delays per day or something, tagged in advance somehow.

szin |previous [-]

[dead]