Introduction: This is not an error, but a ‘resource clogged’ state
Using the generated AI
- It’s crowded
- Please try again after a while
- Not currently available
You may encounter such a display.
In many cases, this is often regarded as an ‘AI bug’ or a ‘cloud failure,’ but in reality,A state that is quite deliberately generatedThat’s right.
Generated AI is not a ‘light API’
The generated AI appears to be answering immediately upon receiving the question.
However, internally, the following processing occurs every time.
- Billions of to billions of parameters
- Multiplication between huge matrices
- Dozens of layers in a row
Like thisA heavy calculation that is almost exclusive to the GPUwill run.
Why a GPU is required
This process is not a realistic speed on the CPU. GPU (Graphics Processing Unit) refers to an image processing device. When generated AI became the mainstream, GPU performance was very suitable for math matric calculations, so it is now used.
The GPU is
- the same command
- to a large amount of data
- Applicable at the same time
I will.
However, it is important thatGPUs like CPUs are ‘bad for complex sequential processing’That’s the point.
GPU is prone to ‘waiting’
Inside the GPU, thousands to tens of thousands of computational units are doing the same processing all at once.
This is fast, but
- Switching between the processing being executed in the middle
- Execute some of the actions of another user
I’m not good at CPU-like timeshare.
Therefore, in the generated AI that uses the GPU,
- One request grabs the GPU
- In the meantime, other requests wait
- The queue grows
It becomes the structure.
What happens when simultaneous access increases
When access is concentrated, the back side will be in the following state.
- All GPUs are in use
- New request to queue
- Wait time exceeds a certain time
- Judging that if you take more than this, the whole thing will collapse
- Reject new requests
This is the state of ‘unusable’ as seen by the user. However, these areControl to prevent collapse, not obstacleThat’s right.
Why can’t you ‘take all of it and then process it later’?
In the generated AI,
- Sequential processing for writing sentences (one token at a time)
- Consumes a lot of GPU memory
- GPU communication also occurs
I will.
Therefore, if you receive unlimited
- GPU memory depleted
- Extremely delayed processing
- Time outs often occur
- As a result, everyone becomes unusable
In other words, stopping at the entrance can minimize the damage.
Physical constraints that don’t disappear even in the cloud
The cloud looks infinite, but the substance is
- GPUs are physically expensive
- There is an upper limit on power and cooling
- The floor space of the data center is also limited.
There are problems like this. This is because the cloud is a virtual server, but it is built on top of the physical server.
The generated AI is‘Software-style gigantic computing infrastructure’and you cannot escape the laws of physics.
‘Stop decision’ is a high-level design decision
In system operation, ‘I’ll take it all, don’t stopis not necessarily the correct answer.
Rather, it is important that
- where to stop
- how long will you wait
- Who do you prioritize
That’s the decision.
The generated AI ‘cannot be used now’ is ‘A control point to protect the limited resources of the GPUThat’s right.
Summary: AI stops not ‘failure’, but a phenomenon designed
The message ‘Unable to use’ or ‘crowded’ displayed on the generated AI screen is not just an error or a failure.
Behind it,
- AI is not an infinite calculation
- The cloud doesn’t have infinite resources
- Everything is based on physical constraints and design decisions
That’s a very realistic situation.
This is not a defect
Control for stabilization intentionally chosen to keep the whole service from destroying itThat’s right.
Generation AI is not magic, it is a technology that is an extension of infrastructure.
Is there anything that the user can do?
Is there anything that the user can do? In conclusion,
This condition is not a disabilityresource controlTherefore, it cannot be ‘fixed’ directly on the user side.
However, there are reasonable actions that can be taken with understanding.
- Avoid busy hours (immediately after the start of work, lunch break, immediately after the event)
- Do not repeat the operation many times immediately after the display appears, wait for a while and try again
- Don’t wastefully change settings or log in again if you decide it’s ‘broken’
- Produced AI is not always a tool that can be responded immediately, so use it with plenty of time to spare.
As long as the resources are limited, there is almost nothing that the user can do. It’s about avoiding congestion and avoiding secondary disasters. When it stops, I think it would be good if I could have the leeway in my mind, ‘Well, AI isn’t perfect,’ instead of thinking, ‘Oh, no more!’
