ARTICLE AD BOX
I am using Django + Celery with Redis as both the broker and the result backend. Here is my Celery configuration:
# Celery settings CELERY_BROKER_URL = 'redis://redis:6379/1' CELERY_RESULT_BACKEND = 'redis://redis:6379/2' CELERY_BROKER_POOL_LIMIT = 10 CELERY_REDIS_MAX_CONNECTIONS = 10 CELERY_BROKER_TRANSPORT_OPTIONS = {'max_connections': 10} CELERY_RESULT_BACKEND_TRANSPORT_OPTIONS = {'max_connections': 10} CELERY_ACCEPT_CONTENT = ['application/json'] CELERY_TASK_SERIALIZER = 'json' CELERY_RESULT_SERIALIZER = 'json' WORKER_MAX_MEMORY_PER_CHILD_KB = 200_000 CELERY_WORKER_MAX_MEMORY_PER_CHILD = 200_000 CELERY_WORKER_MAX_TASKS_PER_CHILD = 1000 CELERY_BEAT_SCHEDULE = celery_beat_schedule CELERY_TASK_DEFAULT_QUEUE = 'default' CELERY_TASK_QUEUES = ( Queue('default'), Queue('bind_status'), ) CELERY_ONCE = { 'backend': 'celery_once.backends.Redis', 'settings': { 'url': "redis://redis:6379/3", 'default_timeout': 60*60, } } app = Celery('') app.config_from_object('django.conf:settings', namespace='CELERY') app.conf.worker_send_task_events = True app.conf.task_send_sent_event = True app.conf.task_track_started = True app.conf.result_extended = True app.conf.broker_connection_retry_on_startup = True app.autodiscover_tasks()I launch my workers like this:
celery -A giberno worker -Q bind_status --loglevel=info -P gevent -c 4Expected Behavior
Given the max_connections=10 settings, I expected the total number of TCP connections to Redis (broker + result backend) to be around 10–20, especially with concurrency=4.
Observed Behavior
When I check the number of connections to Redis DB 2 (used as the result backend), I see hundreds of connections (100–400) even though the `max_connections` limits are set.
Questions
Why does Celery create so many connections despite the max_connections limits?
How can I correctly limit the number of TCP connections when using Redis as the result backend with gevent workers?
Is there a way to make the connection pool actually shared across all greenlets and workers?
