Skip to content

Scraping periodically returns potentially cached results. #46

@FeraruSilviuMarian

Description

@FeraruSilviuMarian

Issue: When posting a new video, scraping returns the correct video id's including the new video, the next scrape returns the previous videos without the new video, the third time scraping returns the correct video ids.

How to reproduce:
Every 70 seconds
videos = scrapetube.get_channel(channel_username = your youtube username)
video_ids = [video['videoId'] for video in videos]
print(video_ids)

Run the script and wait for a print to compare with, then post a video on your youtube channel, and wait for 3+ prints, then compare the results.

I'll use numbers instead of youtube video ID's to demonstrate what results i get, think of the number as a youtube video ID.

[5,4,3,2,1] (state of the channel before new video posted)
[6,5,4,3,2] (new video posted)
[5,4,3,2,1] (scrape now returns the old state of the channel, the previous 5 videos, is this from cache?)
[6,5,4,3,2] (from now on, it returns the correct video ID's)
[6,5,4,3,2]
[6,5,4,3,2]

EDIT: I increased the scrape period to 120 seconds and that worked

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions