Geometry Dash Vision Bot is not jumping at the right time

For this project, my goal is to create a bot that can play Geometry Dash without retrying the game. I am starting step by step, with my first bot trying to jump on the first spike on Stereo Madness. The approach my bot uses is that it takes 2 photos with a 100-millisecond delay between them, to see how much the spike moved, allowing me to calculate the speed of the player. Then, I made the bot take a 3rd photo, and this will check the distance between the player and the spike, allowing it to calculate time to jump from speed. It also adjusts for lag.

This approach is not working however, and my output is this:

PS C:\Users\user\GeoBot> python geodash.py Waiting for loading [(918, 63, 93, 87), (1024, 62, 94, 88)] 633.5490648445683 [(8, 156, 1184, 44), (8, 150, 1184, 2), (945, 77, 40, 57), (221, 68, 79, 79), (1194, 0, 6, 200), (0, 0, 6, 200)] [(945, 77, 40, 57), (221, 68, 79, 79)]

Image link outputs above

When I tried it, it showed a low number of about 600-700 pixels per second. However, when that was run, it did not jump above the spikes.

I tried multiple ways to fix the bug. First, I made sure to print the contours to see if it can detect the right contours, to make sure it does not consider any erratic noise as a part of the speed or distance calculation. As you can see on the output, the contours are fine, and they are accurate.

This led to no difference at all. Then, I made sure to check if the lag was modified, which led to nothing. In the end, I saw that there was something wrong with the speed.

My question is that what is wrong with the speed calculation, and how can I fix it while using the same algorithm?

from selenium import webdriver from selenium.webdriver.support.ui import WebDriverWait from selenium.webdriver.common.by import By from selenium.webdriver.common.action_chains import ActionChains from selenium.webdriver.support import expected_conditions as EC from selenium.webdriver.chrome.options import Options import time import cv2 import numpy as np from pynput.keyboard import Controller, Key import mss # --- Browser Setup --- chrome_options = Options() chrome_options.add_argument("--window-size=1200,1000") chrome_options.add_argument("--force-device-scale-factor=1") driver = webdriver.Chrome(options=chrome_options) # --- Load Game --- driver.get("https://lolygames.github.io/gd-lit/") print("Waiting for loading") time.sleep(8) # --- Click Canvas to Focus, Then Click to Start --- canvas = driver.find_element(By.ID, "#canvas") actions = ActionChains(driver) actions.move_to_element_with_offset(canvas, 0, 0).click().perform() time.sleep(0.5) actions.move_to_element_with_offset(canvas, 0, -100).click().perform() time.sleep(2.5) with mss.mss() as sct: screen_region = {"top": 40, "left": 10, "width": 1200, "height": 1000} # --- Capture Two Frames 0.3s Apart for Speed Calculation --- grab_start = time.time() frame_1_raw = np.array(sct.grab(screen_region)) time.sleep(0.1) frame_2_raw = np.array(sct.grab(screen_region)) grab_elapsed = time.time() - grab_start # --- Crop to Game Play Area --- frame_1_crop = frame_1_raw[600:1000, 0:1200] frame_2_crop = frame_2_raw[600:1000, 0:1200] # --- Diff the Two Frames to Detect Movement --- diff = cv2.absdiff(frame_1_crop, frame_2_crop) diff_gray = cv2.cvtColor(diff, cv2.COLOR_BGR2GRAY) ret, diff_thresh = cv2.threshold(diff_gray, 30, 255, cv2.THRESH_BINARY) # --- Find Bounding Rects of Moving Objects --- contours, hierarchy = cv2.findContours(diff_thresh, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE) moving_rects = [] for cnt in contours: x, y, w, h = cv2.boundingRect(cnt) if w >= 30 and h >= 30: moving_rects.append((x, y, w, h)) cv2.rectangle(diff_thresh, (x, y), (x + w, y + h), (255, 0, 0), 2) print(moving_rects) # --- Calculate Game Speed from Object Displacement --- object_displacement = abs(moving_rects[0][0] - moving_rects[1][0]) game_speed = object_displacement / grab_elapsed print(game_speed) # --- Capture Third Frame to Detect Player + Obstacle Positions --- detect_start = time.time() frame_3_raw = np.array(sct.grab(screen_region)) frame_3_gray = cv2.cvtColor(frame_3_raw, cv2.COLOR_BGR2GRAY) ret, frame_3_thresh = cv2.threshold(frame_3_gray, 30, 255, cv2.THRESH_BINARY_INV) # --- Crop Above Ground Line (Player + Obstacles Only) --- play_area = frame_3_thresh[600:800, :] contours, hierarchy = cv2.findContours(play_area, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE) play_rects = [] allrects=[] for cnt in contours: x, y, w, h = cv2.boundingRect(cnt) allrects.append((x, y, w, h)) cv2.rectangle(play_area, (x, y), (x + w, y + h), (255, 0, 0), 2) if w >= 30 and h >= 30 and w<=200: play_rects.append((x, y, w, h)) # --- Calculate Distance and Jump at the Right Time --- print(allrects) print(play_rects) try: player_obstacle_distance = abs(play_rects[0][0] - play_rects[1][0]) processing_time = time.time() - detect_start jump_delay = (player_obstacle_distance / game_speed) - processing_time print(jump_delay) time.sleep(max(0, jump_delay)) keyboard = Controller() keyboard.press(Key.space) keyboard.release(Key.space) except: print(play_rects) # --- Show Debug Windows --- cv2.imshow('GEODASH CHANGE', diff_thresh) cv2.imshow('GEODASH GAME PHOTO', play_area) cv2.waitKey(0) cv2.destroyAllWindows()