ARTICLE AD BOX
I'm following along this tutorial: https://www.youtube.com/watch?v=PHsC_t0j1dU&t=740s
As you can see in the video (1:29:09) when the person runs docker compose up in his terminal he doesn't get an error (elt-elt_script-1 exited with code 0).
But with mine when I run the command in my terminal I get hit with this:
elt_script-1 exited with code 1
With the following error coming from my python script:
Traceback (most recent call last):
elt_script-1 | File "elt_script.py", line 54, in <module>
elt_script-1 | subprocess.run(dump_command, env=subprocess_env, check=True)
elt_script-1 | File "/usr/local/lib/python3.8/subprocess.py", line 516, in run
elt_script-1 | raise CalledProcessError(retcode, process.args,
elt_script-1 | subprocess.CalledProcessError: Command '['pg_dump', '-h', 'source_postgres', '-U', 'postgres', '-d', 'source_db', '-f', 'data_dump.sql', '-w']' returned non-zero exit status 1.
I've been trying to figure this out for so long and tried to fix this problem for hours now and can't really find a solution now. I'm going to keep trying but I gonna need your help. I'm new to this so it's kinda doing my head in right now.
As we can see from this error that the terminal is giving me running this command it has to do something with my python (elt_script.py) script which I'll paste here:
import subprocess import time def wait_for_postgres(host, max_retries=5, delay_seconds=5): retries = 0 while retries < max_retries: try: result = subprocess.run( ["pg_isready", "-h", host], check=True, capture_output=True, text=True) if "accepting connections" in result.stdout: print("Successfully connected to Postgres") return True except subprocess.CalledProcessError as e: print(f"Error connecting to Postgres: {e}") retries += 1 print( f"Retrying in {delay_seconds} seconds... (Attempt {retries}/{max_retries})") time.sleep(delay_seconds) print("Max retries reached. Exiting.") return False if not wait_for_postgres(host="source_postgres"): exit(1) print("Starting ELT script...") source_config = { 'dbname': 'source_db', 'user': 'postgres', 'password': 'secret', 'host': 'source_postgres' } destination_config = { 'dbname': 'destination_db', 'user': 'postgres', 'password': 'secret', 'host': 'destination_postgres' } dump_command = [ 'pg_dump', '-h', source_config['host'], '-U', source_config['user'], '-d', source_config['dbname'], '-f', 'data_dump.sql', '-w' ] subprocess_env = dict(PGPASSWORD=source_config['password']) subprocess.run(dump_command, env=subprocess_env, check=True) load_command = [ 'psql', '-h', destination_config['host'], '-U', destination_config['user'], '-d', destination_config['dbname'], '-a', '-f', 'data_dump.sql', ] subprocess_env = dict(PGPASSWORD=destination_config['password']) subprocess.run(load_command, env=subprocess_env, check=True) print("Ending ELT Script...")While I'm at it I'll paste my docker.yaml as well as my Dockerfile too if that'll help at all.
I appreciate it for sticking around this long, your help would be much appreciated, constructive feedback also welcome as I want to grow and continue learning. If you need anymore information please don't hesitate to ask any questions.
Dockerfile:
FROM python:3.8-slim
RUN apt-get update && apt-get install -y postgresql-client
COPY elt_script.py .
CMD [ "python", "elt_script.py" ]
docker-compose.yaml:
version: '3' services: source_postgres: image: postgres:18 ports: - "5433:5432" networks: - elt_network environment: POSTGRES_DB: source_db POSTGRES_USER: postgres POSTGRES_PASSWORD: secret volumes: - ./source_db_init/init.sql:/docker-entrypoint-initdb.d/init.sql destination_postgres: image: postgres:18 ports: - "5434:5432" networks: - elt_network environment: POSTGRES_DB: destination_db POSTGRES_USER: postgres POSTGRES_PASSWORD: secret volumes: - ./source_db_init/init.sql:/docker-entrypoint-initdb.d/init.sql elt_script: build: context: ./elt dockerfile: Dockerfile command: ["python", "elt_script.py"] networks: - elt_network depends_on: - source_postgres - destination_postgres networks: elt_network: driver: bridge