How to paginate and process large JSON API responses in Python without hitting errors or duplicates?

I am working with a REST API that returns a large JSON dataset (hundreds of thousands of records). The API supports pagination using parameters like startIndex and resultsPerPage.

I am using Python with the requests library. I can fetch small amounts of data and process it correctly.

However, I am unsure how to handle the full dataset efficiently and safely.

Here is a simplified version of my current code:

This works for small datasets, but I am not sure how to scale it for large responses.

My questions are:

1. What is the correct way to loop through all pages using parameters like startIndex?
2. How can I handle cases where response.json() fails (for example JSONDecodeError)?
3. What is a good approach to avoid duplicate records when storing data?
4. Are there best practices for handling large API responses in Python?

Any guidance would be helpful

import requests url = "https://example.com/api?resultsPerPage=5" response = requests.get(url) if response.status_code == 200: data = response.json() for item in data["items"]: print(item["id"]) else: print("Error:", response.status_code)