The 11 best web scraping subreddits

Web scraping is an essential skill for data analysts and developers who want to extract data from websites. However, finding reliable sources to learn and discuss web scraping techniques can be challenging. Fortunately, several subreddits on Reddit are dedicated to web scraping, data analysis, and programming-related discussion.

In this article, we'll explore the 11 best subreddits for web scraping and share why each of these subreddits might be useful for you on your web scraping journey.

Here is the list of subreddits we will be focusing on:

• r/WebScraping
• r/Scrapy
• r/Selenium
• r/Puppeteer
• r/ProxiesTalk
• r/ProxyLists
• r/ShoeBots
• r/dataisbeautiful
• r/learnprogramming
• r/datanalysis
• r/dataengineering

r/WebScraping

Statistics:

• Total subscribers: 12,329
• Posts per day: 4
• Comments per day: 7
• Growth in subscribers (since last year): 4,812

The first one on our list is r/WebScraping . This subreddit is very useful for keeping up to date with the latest news in the world of web scraping. If you ever have any trouble with scraping a website, this should be the first place you look for help. The community is generally very responsive and supportive, and members are often willing to offer guidance or share their solutions. You will find novel ideas shared here as well. For example, this GPT-powered web scraper!

r/Scrapy

Statistics:

• Total subscribers: 5,529
• Posts per day: 1
• Comments per day: 2
• Growth in subscribers (since last year): 733

Scrapy is a very powerful, robust, and mature Python web scraping framework used by companies of all sizes. The /r/scrapy subreddit is dedicated to all things Scrapy. It is the official community hub of Scrapy and so you will often find Scrapy experts hanging out there.

If you encounter an issue or have questions related to Scrapy, this subreddit is an excellent resource to seek help and guidance.

r/Selenium

Statistics:

• Total subscribers: 10,955
• Posts per day: 3
• Comments per day: 5
• Growth in subscribers (since last year): 1,777

No matter where you are on your web scraping journey, chances are that you will have heard of Selenium at one point. It is a framework for browser automation and is an indispensable tool for scraping experts. The /r/selenium subreddit will help you learn how to effectively use this powerful tool. You can ask all sorts of questions regarding Selenium. For example, a recent question was asking about what else prevents finding elements besides IFrames in Selenium.

r/Puppeteer

Statistics:

• Total subscribers: 430
• Posts per day: 1
• Comments per day: 3
• Growth in subscribers (since last year): 138

Puppeteer is a popular browser automation library maintained by Google. It is an alternative to Selenium and Playwright . The /r/puppeteer subreddit is not very active but you can find solutions to Puppeteer-related issues and Puppeteer-specific tips in old posts. You might be able to get help by making a new post but we wouldn't suggest you count on it much as as only approved members can make posts.

r/ProxiesTalk

Statistics:

• Total subscribers: 195
• Posts per day: 1
• Comments per day: 1
• Growth in subscribers (since last year): 104

Good reliable proxies are always a necessity when doing any kind of serious web scraping. You don't always have to rely on paid proxies. You can use subreddits like /r/ProxiesTalk to ask proxy-related questions and potentially source some cheap, low-cost proxies.

r/ProxyLists

Statistics:

• Total subscribers: 468
• Posts per day: 4
• Comments per day: 2
• Growth in subscribers (since last year): 248

/r/ProxyLists subreddit is similar to /r/ProxiesTalk . It is dedicated to sharing free proxies. However, you might also find posts by companies pushing their paid proxy lists. If you have time and can sift through the noise, you can score some cheap (free) and reliable proxies in this subreddit.

r/ShoeBots

Statistics:

• Total subscribers: 40,981
• Posts per day: 3
• Comments per day: 7
• Growth in subscribers (since last year): 2,694

Shoe scalpers very often use bots for automated purchases on shoe websites like Nike and Adidas . These websites often have the most stringent anti-bot measures to thwart scalping and automated buying efforts. The /r/ShoeBots subreddit is a superb resource for exploring the scalping and shoebots side of web scraping. The posts in this subreddit might give you some clever ideas to bypass some of the strongest anti-bot measures in the industry. We have also covered some anti-bot evasion techniques in the past.

r/dataisbeautiful

Statistics:

• Total subscribers: 19,368,352
• Posts per day: 39
• Comments per day: 2292
• Growth in subscribers (since last year): 2,374,536

If you are interested in web scraping from the perspective of data analysis, the /r/dataisbeautiful subreddit will give you a ton of fun visualization ideas. It is an extremely active community with over 39 posts made every day and quite a few of them regularly make it to Reddit's front page. You also don't need to be interested in data analysis to be mesmerized by the engaging visualizations shared by the /r/dataisbeautiful community.

r/learnprogramming

Statistics:

• Total subscribers: 3,731,058
• Posts per day: 131
• Comments per day: 434
• Growth in subscribers (since last year): 1,343,900

Any serious web scraping requires getting your hands dirty with some coding. Oftentimes your scraping questions can more appropriately be defined as coding/programming questions. For such questions, /r/learnprogramming is the best place to seek help. It is a very welcoming community where beginners are encouraged to ask all kinds of programming-related questions. With the huge amount of active members, most posts get a response very quickly.

r/dataanalysis

Statistics:

• Total subscribers: 47,494
• Posts per day: 26
• Comments per day: 43
• Growth in subscribers (since last year): 26,175

/r/dataanalysis is similar to /r/dataisbeautiful . However, this subreddit is targeted more toward asking questions, sourcing feedback, and sharing tips and tricks regarding data analysis. You can ask data analysis-related questions in this subreddit and once your analysis is complete, you can share a visualization in the /r/dataisbeautiful subreddit.

r/dataengineering

Statistics:

• Total subscribers: 91,858
• Posts per day: 40
• Comments per day: 173
• Growth in subscribers (since last year): 39,381

Gathering data is just one part of web scraping; storing that data is a whole other beast. If you have a ton of data and need help, guidance, or suggestions related to different data storage options and data pipelines, the /r/dataengineering subreddit is your best friend. Ask whichever question you might have related to data engineering and you will get a response pretty quickly. Quite a few veterans with data engineering experience hang out here.

Recap

Subreddit	Total Subscribers	Posts per day	Comments per day	Growth in subscribers (since last year)
r/WebScraping	12,329	4	7	4,812
r/Scrapy	5,529	1	2	733
r/Selenium	10,955	3	5	1,777
r/Puppeteer	430	1	3	138
r/ProxiesTalk	195	1	1	104
r/ProxyLists	468	4	2	248
r/ShoeBots	40,981	3	7	2,694
r/dataisbeautiful	368,352	39	2292	2,374,536
r/learnprogramming	731,058	131	434	1,343,900
r/dataanalysis	47,494	26	43	26,175
r/dataengineering	91,858	40	173	39,381

Conclusion

This was a short roundup of some of the best subreddits we found related to web scraping. We hope that you use these resources to improve your knowledge and take your web scraping expertise to the next level. If you have any scraping-related questions that are not covered by any of these subreddits, you can browse through our blog which is choke-full of informative posts.