Last night, >20% of Reddit links on Google weren't reachable
Insomnia strikes again!
I wrote a script to fetch Reddit links shown on the first page of Google search results and check whether the subreddit linked was private or not.
I asked ChatGPT to generate a sample of 100 common Google searches and appended "reddit" to the end of them.
Results of Survey
. total private percentage term
0 10 4 40.0 django or ruby reddit
1 10 3 30.0 best chair reddit
2 9 0 0.0 funny meme reddit
3 10 0 0.0 reverse string reddit
4 10 6 60.0 iOS or Android reddit
5 10 4 40.0 best laptop for coding reddit
6 10 2 20.0 Python or Java reddit
7 10 4 40.0 web development framework comparison reddit
8 5 5 100.0 best budget headphones reddit
9 10 0 0.0 programming language comparison reddit
10 10 4 40.0 JavaScript or TypeScript reddit
11 10 0 0.0 best travel destinations reddit
12 12 0 0.0 book recommendations reddit
13 10 0 0.0 best online coding courses reddit
14 10 4 40.0 fitness tips for beginners reddit
15 10 1 10.0 WordPress or Drupal reddit
16 10 2 20.0 best gaming mouse reddit
17 5 3 60.0 top programming subreddits
18 10 0 0.0 best coding bootcamps reddit
19 9 4 44.44 career advice for developers reddit
20 5 4 80.0 best budget smartphone reddit
21 10 1 10.0 comparison between Mac and Windows reddit
22 10 0 0.0 best hiking trails reddit
23 9 0 0.0 dieting tips reddit
24 10 2 20.0 React or Vue reddit
25 9 0 0.0 best podcast recommendations reddit
26 10 0 0.0 self-improvement tips reddit
27 10 0 0.0 best coffee brewing methods reddit
28 11 0 0.0 fiction book recommendations reddit
29 10 6 60.0 DIY home improvement projects reddit
30 5 3 60.0 best budget DSLR camera reddit
31 10 1 10.0 traveling with pets tips reddit
32 10 0 0.0 best vegetarian recipes reddit
33 10 1 10.0 coding bootcamp reviews reddit
34 10 1 10.0 top cities for remote work reddit
35 10 0 0.0 best board games for game nights reddit
36 10 1 10.0 music production software comparison reddit
37 7 1 14.29 home workout routines reddit
38 5 0 0.0 best budget travel destinations reddit
39 10 0 0.0 motorcycle vs car for commuting reddit
40 10 0 0.0 best productivity apps reddit
41 4 0 0.0 cryptocurrency investment advice reddit
42 10 10 100.0 anime recommendations reddit
43 10 0 0.0 best hiking gear reddit
44 10 6 60.0 remote job interview tips reddit
45 10 1 10.0 coding challenges for beginners reddit
46 10 0 0.0 plant-based diet tips reddit
47 10 5 50.0 best gaming keyboards reddit
48 10 1 10.0 freelancing tips and tricks reddit
49 14 0 0.0 foreign language learning resources reddit
50 5 1 20.0 budget-friendly home decor ideas reddit
51 5 5 100.0 best budget wireless headphones reddit
52 13 0 0.0 healthy meal prep ideas reddit
53 10 8 80.0 photography equipment recommendations reddit
54 10 0 0.0 web design vs web development reddit
55 10 0 0.0 book suggestions for self-improvement reddit
56 10 0 0.0 creative writing tips reddit
57 5 1 20.0 best budget fitness trackers reddit
58 10 3 30.0 advice for first-time homebuyers reddit
59 10 1 10.0 top meditation apps reddit
60 10 0 0.0 best hiking trails near me reddit
61 9 0 0.0 cooking tips and tricks reddit
62 10 1 10.0 programming languages comparison reddit
63 5 4 80.0 best budget-friendly smartphones reddit
64 10 4 40.0 self-care routine ideas reddit
65 10 1 10.0 best podcasts for personal development reddit
66 8 0 0.0 home office setup inspiration reddit
67 10 5 50.0 top indie games recommendations reddit
68 10 0 0.0 tips for starting a small business reddit
69 5 0 0.0 best budget-friendly laptops for students reddit
70 10 3 30.0 fitness motivation and tips reddit
71 9 0 0.0 gardening hacks and tips reddit
72 10 0 0.0 best sci-fi book series reddit
73 5 1 20.0 budget-friendly meal planning reddit
74 10 7 70.0 tips for mastering a musical instrument reddit
75 10 0 0.0 career change advice reddit
76 5 3 60.0 best budget-friendly home workout equipment reddit
77 10 7 70.0 professional networking tips reddit
78 4 0 0.0 top documentaries to watch reddit
79 10 9 90.0 outdoor photography tips and tricks reddit
80 5 0 0.0 best budget-friendly skincare products reddit
81 10 10 100.0 yoga for beginners resources reddit
82 10 0 0.0 cooking with limited ingredients recipes reddit
83 9 2 22.22 top DIY craft ideas reddit
84 10 2 20.0 car buying tips and negotiation strategies reddit
85 9 0 0.0 best personal finance books reddit
86 10 2 20.0 coding bootcamp success stories reddit
87 10 0 0.0 tips for improving focus and concentration reddit
88 5 5 100.0 home renovation ideas on a budget reddit
89 10 4 40.0 top podcasts for true crime enthusiasts reddit
90 10 0 0.0 beginner-friendly painting techniques reddit
91 5 0 0.0 best budget-friendly wireless routers reddit
92 10 7 70.0 tips for sustainable living reddit
93 10 2 20.0 finding remote freelance gigs reddit
94 10 0 0.0 book recommendations for historical fiction lovers reddit
95 5 0 0.0 budget-friendly travel gear reddit
96 5 3 60.0 best budget-friendly noise-canceling headphones reddit
97 9 3 33.33 tips for improving sleep quality reddit
98 10 2 20.0 italy hotels reddit
Taking the mean of the percentages column, we get 23.8%. Potentially over a fifth of Reddit links being served to users are blocked behind private subreddits.
Think or do whatever you wish with this info.
The script
import urllib.parse
import requests
cache = {}
def search_and_check_reddit():
with open("terms.txt", "r") as f:
for term in f.readlines():
headers = {
"user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) "
}
page = requests.get(f"https://www.google.com/search?q={term}", headers=headers)
links = [link.split('&')[0][7:] for link in page.text.split('href="')[1:]]
links = [link for link in links if link[:4] == "http" and "reddit.com" in link]
total = len(links)
private = 0
for link in links:
parsed_link = urllib.parse.urlparse(link)
subreddit = parsed_link.path.split('/')[2]
if subreddit in cache:
if cache[subreddit] is True:
private += 1
continue
response = requests.get(link, allow_redirects=True)
if "private-community-modal" in response.text:
cache[subreddit] = True
private += 1
else:
cache[subreddit] = False
print(f"{total}\t{private}\t{private/total*100:.2f}%\t{term}")
search_and_check_reddit()