Submissions SubredditStream at times fetches old reddit posts despite using `skip_existing`

[Bronya-Rand]

Describe the Bug

I was upgrading the codebase of a Discord bot that used Asyncpraw for a long while to obtain posts, however I've begun to notice that the code that I had originally and the new code now have begun to break recently where, at certain times, the submission stream just begins to fetch old posts as if they were "new" causing the bot to post those old posts as if they were newly posted.

It seems the logic in skip_existing no longer works as of late? I tried searching in the r/redditdev subreddit and only a deleted post seems to refer to this issue.

Desired Result

SubmissionStream should only be fetching posts that are newly posted on a given subreddit.

Code to reproduce the bug

class Reddit(commands.Cog):
    def __init__(self, bot):
        self.bot: commands.Bot = bot # Discord.py
        self.reddit_session: asyncpraw.Reddit
        self.reddit_channel: TextChannel # Discord.py

    async def fetch_subreddit_posts(self):
        subreddit = await self.reddit_session.subreddit("arandomsubreddit")
        logger.info("Fetching new subreddit posts...")
        try:
            async for submission in subreddit.stream.submissions(
                skip_existing=True
            ):
                embed_type = "link"
                url_lower = submission.url.lower()
                if url_lower.endswith((".jpg", ".jpeg", ".png", ".webp")):
                    embed_type = "image"
                elif url_lower.endswith(".gif"):
                    embed_type = "gifv"
                elif "youtu" in url_lower or "vimeo" in url_lower:
                    embed_type = "video"
                elif submission.is_self:
                    embed_type = "rich"

                if "reddit.com/gallery" in url_lower:
                    embed_type = "link"  # Special handling for galleries

                embed = create_submission_embed(submission, embed_type)

                await self.reddit_channel.send(embed=embed)
        except (ServerError, RequestException, Forbidden) as e:
            logger.error(f"Error fetching subreddit posts: {e}")
        except Exception as e:
            logger.exception(f"Unexpected error: {e}")

My code does not include sensitive credentials

• Yes, I have removed sensitive credentials from my code.

Relevant Logs

There isn't an error in the log file to comment here.

This code has previously worked as intended

Yes

Operating System/Environment

Debian 13

Python Version

3.11.2

Async PRAW Version

7.8.1

Links, references, and/or additional comments?

https://www.reddit.com/r/redditdev/comments/1nzssc3/old_submissions_sporadically_showing_up_as_new/
https://www.reddit.com/r/redditdev/comments/1nrpyum/bot_responding_to_old_posts/

[LilSpazJoekp]

Transferring to PRAW since this issue is not specific to Async PRAW.

[dkozinn]

I've been having the same issue. After discussions with the developers, I changed the value of BoundedSet in models/util.py to 1500 (seen_attributes = BoundedSet(1500) ) to increase the cache size, the idea being that too many items were being returned and the cache couldn't hold enough to prevent the issue. Unfortunately, this did not restult in any improvement.

I am running a test version that logs all requests & responses but have not yet been able to reproduce the issue with this version.

An example of the code that reproduces this can be found at https://github.com/dkozinn/r-nasabot/blob/main/src/nasaxpost.py.

In adding to the 2 links from Reddit above, here's an additional link where the Reddit admins are participating: https://www.reddit.com/r/bugs/comments/1nzpf87/praw_api_old_submissions_sporadically_showing_up/

[LilSpazJoekp]

More discussion is in our slack.

@dkozinn would you mind trying this code:

import json

from prawcore import Requestor

import praw

MAX_HISTORY = 50


class BoundedList(list):
    def __init__(self, max_size):
        super().__init__()
        self.max_size = max_size

    def append(self, item):
        if len(self) >= self.max_size:
            self.pop(0)
        super().append(item)


class DebugRequestor(Requestor):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self._data = BoundedList(MAX_HISTORY)

    def request(self, *args, **kwargs):
        response = super().request(*args, **kwargs)
        # skip recording token requests
        method, url = args
        if (
            args[0] == "post"
            and args[1] == "https://www.reddit.com/api/v1/access_token"
        ):
            return response

        # Create copies to avoid modifying the originals
        _kwargs = kwargs.copy()
        # Remove the headers to avoid logging sensitive info
        _kwargs.pop("headers", None)
        self._data.append(
            {'args': args, 'kwargs': _kwargs, 'response_json': response.json()}
        )
        return response


if __name__ == "__main__":
    reddit = praw.Reddit(..., requestor_class=DebugRequestor)

    previous_item = None
    seen_items = set()

    for item in reddit.subreddit("all").stream.submissions(skip_existing=True):
        if (previous_item is not None and item.created_utc < previous_item.created_utc) or item.id in seen_items:
            print("Out of order items detected")
            # save past requests for inspection
            with open("request_log.json", "w") as f:
                json.dump(reddit._core._requestor._data, f, indent=2)
        seen_items.add(item.id)
        previous_item = item
       

[dkozinn]

I ran it and almost immediately started getting the Out of Order messages. Here's the log, let me know if you need me to run this longer.

request_log.json

[dkozinn]

One of the test versions of my bot that was capturing all the requests encountered the issue a couple of times today. The log file is 313M compressed. Do you want to take a look at it @LilSpazJoekp ? I don't know if I can upload here but I can make it available to you.

For reference the issue first occurred at 11/2/26 09:18:14 GMT then another at 12:35:06 GMT.

[LilSpazJoekp]

Seeing older posts is likely normal since it can happen with approval from modqueue. I would try this code instead:

import json

from prawcore import Requestor

import praw

MAX_HISTORY = 150
SEEN_SIZE = 1500


class BoundedList(list):
    def __init__(self, max_size):
        super().__init__()
        self.max_size = max_size

    def append(self, item):
        if len(self) >= self.max_size:
            self.pop(0)
        super().append(item)


class DebugRequestor(Requestor):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self._data = BoundedList(MAX_HISTORY)

    def request(self, *args, **kwargs):
        response = super().request(*args, **kwargs)
        # skip recording token requests
        method, url = args
        if (
            args[0] == "post"
            and args[1] == "https://www.reddit.com/api/v1/access_token"
        ):
            return response

        # Create copies to avoid modifying the originals
        _kwargs = kwargs.copy()
        # Remove the headers to avoid logging sensitive info
        _kwargs.pop("headers", None)
        self._data.append(
            {'args': args, 'kwargs': _kwargs, 'response_json': response.json()}
        )
        return response


if __name__ == "__main__":
    reddit = praw.Reddit(..., requestor_class=DebugRequestor)

    seen_items = BoundedList(SEEN_SIZE)

    for item in reddit.subreddit("all").stream.submissions(skip_existing=True):
        if item.id in seen_items:
            print("Duplicate seen")
            # save past requests for inspection
            with open("request_log.json", "w") as f:
                json.dump(reddit._core._requestor._data, f, indent=2)
        seen_items.append(item.id)

Is the issue you're referring to today an error with seeing a repeat item? If so, then yeah I'd like to take a look. That is pretty large of a log file but you could try attaching it. If that doesn't work the we can figure something else out.

[dkozinn]

Today's issue is with the repeated (old) items that were definitely not released from modqueue. I'll DM you with the location of the file.

I'll try that other code and get back to you.

[dkozinn]

I got this error after running the above code for a bit, no log file written:

Traceback (most recent call last):
  File "/home/david/.local/lib/python3.10/site-packages/requests/models.py", line 974, in json
    return complexjson.loads(self.text, **kwargs)
  File "/home/david/.local/lib/python3.10/site-packages/simplejson/__init__.py", line 514, in loads
    return _default_decoder.decode(s)
  File "/home/david/.local/lib/python3.10/site-packages/simplejson/decoder.py", line 386, in decode
    obj, end = self.raw_decode(s)
  File "/home/david/.local/lib/python3.10/site-packages/simplejson/decoder.py", line 416, in raw_decode
    return self.scan_once(s, idx=_w(s, idx).end())
simplejson.errors.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/david/src/r-nasabot/src/test_2.py", line 57, in <module>
    for item in reddit.subreddit("all").stream.submissions(skip_existing=True):
  File "/home/david/.local/lib/python3.10/site-packages/praw/models/util.py", line 150, in stream_generator
    for item in reversed(list(function(limit=limit, **function_kwargs))):
  File "/home/david/.local/lib/python3.10/site-packages/praw/models/listing/generator.py", line 66, in __next__
    self._next_batch()
  File "/home/david/.local/lib/python3.10/site-packages/praw/models/listing/generator.py", line 90, in _next_batch
    self._listing = self._reddit.get(self.url, params=self.params)
  File "/home/david/.local/lib/python3.10/site-packages/praw/util/deprecate_args.py", line 46, in wrapped
    return func(**dict(zip(_old_args, args)), **kwargs)
  File "/home/david/.local/lib/python3.10/site-packages/praw/reddit.py", line 731, in get
    return self._objectify_request(method="GET", params=params, path=path)
  File "/home/david/.local/lib/python3.10/site-packages/praw/reddit.py", line 514, in _objectify_request
    self.request(
  File "/home/david/.local/lib/python3.10/site-packages/praw/util/deprecate_args.py", line 46, in wrapped
    return func(**dict(zip(_old_args, args)), **kwargs)
  File "/home/david/.local/lib/python3.10/site-packages/praw/reddit.py", line 963, in request
    return self._core.request(
  File "/home/david/.local/lib/python3.10/site-packages/prawcore/sessions.py", line 328, in request
    return self._request_with_retries(
  File "/home/david/.local/lib/python3.10/site-packages/prawcore/sessions.py", line 234, in _request_with_retries
    response, saved_exception = self._make_request(
  File "/home/david/.local/lib/python3.10/site-packages/prawcore/sessions.py", line 186, in _make_request
    response = self._rate_limiter.call(
  File "/home/david/.local/lib/python3.10/site-packages/prawcore/rate_limit.py", line 47, in call
    response = request_function(*args, **kwargs)
  File "/home/david/src/r-nasabot/src/test_2.py", line 42, in request
    {'args': args, 'kwargs': _kwargs, 'response_json': response.json()}
  File "/home/david/.local/lib/python3.10/site-packages/requests/models.py", line 978, in json
    raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

I'm rerunning in a clean venv now with only praw installed via pip in case there was something else causing the issue.

[LilSpazJoekp]

Let's use Betamax instead. Just pass in the Reddit instance and wrap the stream function call with the .stream() method.

import json
import os
from base64 import b64encode

from urllib.parse import quote_plus

import requests
from betamax import cassette
from betamax import Betamax

import praw

PRAW_CONFIG_SECTION = "nasapostbot"
REQUEST_LIMIT = 15
SEEN_SIZE = 1500


class BoundedList(list):
    def __init__(self, max_size):
        super().__init__()
        self.max_size = max_size

    def append(self, item):
        if len(self) >= self.max_size:
            self.pop(0)
        super().append(item)


class PRAWDebugger:
    def __init__(self, reddit: praw.Reddit):
        self._cassette_index = 0
        self._session = requests.Session()
        self._session.headers["Accept-Encoding"] = "identity"
        self._reddit = reddit
        self._seen_items = BoundedList(SEEN_SIZE)
        self._ensure_cassette_dir()
        self.recorder = Betamax(self._session)
        with Betamax.configure() as config:
            config.before_record(callback=self._filter_access_token)
            config.before_record(callback=self._ensure_interaction_limit)
            config.cassette_library_dir = "cassettes"
            config.default_cassette_options["record_mode"] = "all"

            for key, value in {
                k: getattr(self._reddit.config, k)
                for k in [
                    "client_id",
                    "client_secret",
                    "password",
                    "refresh_token",
                    "username",
                ]
            }.items():
                if key == "password" and value:
                    value = quote_plus(value)
                config.define_cassette_placeholder(f"<{key.upper()}>", key)
            config.define_cassette_placeholder(
                "<BASIC_AUTH>",
                b64encode(
                    f"{self._reddit.config.client_id}:{self._reddit.config.client_secret}".encode()
                ).decode("utf-8"),
            )

    @staticmethod
    def _filter_access_token(interaction, current_cassette):
        """Add Betamax placeholder to filter access token."""
        response = interaction.data["response"]
        if response["status"]["code"] != 200:
            return
        try:
            body = response["body"]["string"]
            token = json.loads(body)["access_token"]
        except (KeyError, TypeError, ValueError):
            return
        current_cassette.placeholders.append(
            cassette.cassette.Placeholder(placeholder="<ACCESS_TOKEN>", replace=token)
        )

    @staticmethod
    def _ensure_interaction_limit(interaction, current_cassette):
        """Ensure that the number of interactions does not exceed REQUEST_LIMIT."""
        if len(current_cassette.interactions) >= REQUEST_LIMIT:
            current_cassette.interactions.pop(0)

    @staticmethod
    def _ensure_cassette_dir():
        if not os.path.exists("cassettes"):
            os.mkdir("cassettes")

    @property
    def reddit(self):
        all_settings = self._reddit.config.__dict__
        custom_settings = all_settings.pop("custom", {})
        remaining_settings = all_settings.pop("_settings", {})
        return praw.Reddit(
            requestor_kwargs={"session": self._session},
            **all_settings,
            **custom_settings,
            **remaining_settings,
        )

    @property
    def cassette_index(self):
        if self._cassette_index == 0:
            try:
                with open(os.path.join("cassettes", ".cassette_index"), "r") as f:
                    self._cassette_index = int(f.read().strip())
            except (FileNotFoundError, ValueError):
                self._cassette_index = 0
        return self._cassette_index

    @cassette_index.setter
    def cassette_index(self, value):
        self._cassette_index = value
        with open(os.path.join("cassettes", ".cassette_index"), "w") as f:
            f.write(str(value))

    def start(self):
        self.cassette_index += 1
        self.recorder.use_cassette(f"interactions_{self.cassette_index}")
        self.recorder.start()

    def stop(self):
        self.recorder.stop()

    def stream(self, func):
        self.start()
        for i in func:
            if i.id in self._seen_items:
                print("Out of order items detected")
                self.stop()
                self.start()
            self._seen_items.append(i.id)
            yield i


if __name__ == "__main__":
    debugger = PRAWDebugger(praw.Reddit(PRAW_CONFIG_SECTION))

    reddit = debugger.reddit

    for item in debugger.stream(
        reddit.subreddit("all").stream.submissions(skip_existing=True)
    ):
        ...
        # Process the item

[dkozinn]

The problem recurred and I've got the Betamax capture. Note that PRAWDebugger didn't "notice" the old posts, the output is from when I stopped it after seeing the issue in the log. The first "bad" post came in at 11/6/24 04:37:27 GMT with a post created on Fri Apr 12 16:43:47 2024. As discussed in Slack, since these are posts to a u/ stream, they wouldn't have all suddenly been re-released from a mod queue.

|Thu Nov 6 04:37:27 2025 WARNING - nasaxpost:main:62 — Did not re-crosspost 'Applications close for NASA's fall+
2024 internships at 11:59pm ET on Friday, April 12' from nasa at https://reddit.com/r/u_nasa/comments/1c2dwkj/a+
pplications_close_for_nasas_fall_2024/ created on Fri Apr 12 16:43:47 2024

interactions_1.json.gz

[dkozinn]

I finally got the notification from PRAW Debugger that it got out of order items. The Betamax cassettes are in the tgz and my app output is in the xpost_logs.gz file to potentially help in finding timestamp, etc.

For some reason, I'm unable to upload the log files, I'll get them to you privately.

[github-actions]

This issue is stale because it has been open for 30 days with no activity.

Remove the Stale label or comment or this will be closed in 30 days.

[dkozinn]

We are waiting for feedback from the Reddit admin team about this.