ARTICLE AD BOX
I am building a backend system using FastAPI where I need to process CPU-intensive tasks such as file scanning and data analysis.
Currently, the processing runs inside the API endpoint, which causes the request to block and significantly slows down the system when handling multiple tasks.
I tried using async endpoints, but performance did not improve since the workload is CPU-bound rather than I/O-bound.
I am considering different approaches such as:
- Celery with Redis or RabbitMQ
- Python multiprocessing
- FastAPI BackgroundTasks
My goal is to design a scalable and production-ready architecture where:
- The API remains responsive
- Tasks can run in parallel
- The system can scale as workload increases
What is the recommended architecture for handling CPU-bound background tasks in FastAPI?
