ARTICLE AD BOX
I run a webserver on my local network to host my movie collection. I also have a custom android app running on an amazon fire stick. Recently I have been working to enable video playback from external sources.
The app sends requests to the webserver, like /externalsearch.php?title=(movie). The site then calls a selenium python web scraper and returns various information in the form of json. All of this works perfectly, about 4 or 5 times. After that, only requests to pure html pages seem to work from the app. The php pages all return 404. Other static files, like /catalog.json also return a 404. Restarting the system gets you another 4 or 5 queries before breaking again.
When visiting the same pages on my computer, they all function perfectly no matter what. However, visiting the pages on my iPhone causes the same result. The only other things running on the
Here is some addition information and things I have already checked. Memory usage is not abnormally high. The app is properly disconnecting. The issue persisted through a completely fresh Debian install on the server. The only other services running on the server are ssh to control it and an smb file share into the web page directory.
Below is one of the kotlin functions that calls the page, the webpage externalsearch.php and the associated python file, with some of the specifics obscurated.
android function
fun external(view : View) { val searchStringLower = searchString.lowercase() CoroutineScope(Dispatchers.IO).launch { var connection: HttpURLConnection? = null try { val urlString = "${MainActivity.settings.ip}/externalsearch.php?title=${URLEncoder.encode(searchStringLower, "UTF-8")}" val url = URL(urlString) connection = url.openConnection() as HttpURLConnection connection.requestMethod = "GET" connection.connectTimeout = 5000 connection.readTimeout = 15000 val responseCode = connection.responseCode if (responseCode == HttpURLConnection.HTTP_OK) { val response = connection.inputStream.bufferedReader().use { it.readText() } val json = Json { ignoreUnknownKeys = true } val results: List<ExternalResult> = json.decodeFromString(response) lifecycleScope.launch(Dispatchers.Main) { val adapter = ExternalItemAdapter(this@SearchActivity, results, 0) searchGrid.adapter = adapter } } else { println("Error: $responseCode") } } catch (e: Exception) { e.printStackTrace() } finally { connection?.disconnect() } } }externalsearch.php
<?php if (!isset($_GET['title'])) { echo json_encode(["error" => "Missing title parameter"]); exit; } // Escape argument to prevent command injection $title = escapeshellarg($_GET['title']); // Local script reference $script = __DIR__ . "/Scripts/externalsearch.py"; // Build command and execute $cmd = "xvfb-run -a python3 $script $title 2>&1"; $output = shell_exec($cmd); header("Content-Type: application/json"); echo $output; ?>/Scripts/externalsearch.py
import sys import json import re from selenium import webdriver from selenium.webdriver.common.by import By from selenium.webdriver.support.ui import WebDriverWait from selenium.webdriver.support import expected_conditions as EC from selenium.webdriver.firefox.service import Service from bs4 import BeautifulSoup from threading import Lock if len(sys.argv) < 2: print(json.dumps({"error": "Missing title parameter"})) sys.exit(1) title = sys.argv[1] url = f"(url)" options = webdriver.FirefoxOptions() options.add_argument("--headless") service = Service('/usr/local/bin/geckodriver') driver = webdriver.Firefox(service=service, options=options) results = [] try: driver.get(url) html = driver.page_source soup = BeautifulSoup(html, "html.parser") ... Scraping code ... except Exception as e: print(json.dumps({"error": str(e)})) finally: driver.quit() print(json.dumps(results))I'm not entirely sure which part of the stack is at fault for this issue, so I have not included any logs. If you think something additional would be helpful please let me know.
