Yes i do, i like to put down my thoughts in writing as well as planning out strategies and such as i found out that i could not find a pen when i needed it much which was a dilemma of mine and so i started just using my laptop and phone to jot down what came to mind and was worth recording
50 years of one 50 word grocery list per week (if you keep them) is 130,000 words, but not everyone writes that much, or that consistently, or for that long, or keeps complete digital copies.
My “Ask HN” was not intended to focus on HN comment writing only, but since it headed that way on its own, can you share a scriptlet so that others may do more than speculate?
eth0up your last comment went auto-[dead] probably because of the python. I vouched for it, think that much worked.
I would just grab it and add to a text file URL but python cares about spaces and the script is not consistently indented 4 spaces to make it into HN code. Do you by chance have a git repo you can save it to?
Another option is to sftp it to hn@nochan.net (no password), put in /pub/ and I can turn it into a URL. If you can reach it.
If you end up truly at a dead end, let me know and I'll go the sftp route. But if you would, please try dropping the rough paste into GPT, Claude, Gemini, etc, and just ask it to clean it up. I don't think there are any missing lines. If that fails let me know, but I generally avoid direct IP contact with strangers, and I am kind of not crazy about git and most things.
It's ok, I was just trying to help cwmoore get the script but it looks like they already play with LLM's so I will let them figure it out from what you pasted. Thanks for contributing your script.
def fetch_page(username, page, created_after=None, created_before=None):
params = {
"tags": f"comment,author_{username}",
"hitsPerPage": HITS_PER_PAGE,
"page": page,
}
filters = []
if created_after is not None:
filters.append(f"created_at_i>{created_after}")
if created_before is not None:
filters.append(f"created_at_i<{created_before}")
if filters:
params["numericFilters"] = ",".join(filters)
def get_total_in_range(username, created_after=None, created_before=None):
data = fetch_page(username, 0, created_after, created_before)
return data["nbHits"], data
def dump_range(username, created_after, created_before, seen_ids, all_records, depth=0):
"""Recursively bisect time ranges to stay under Algolia's 10k page wall."""
total, first_page = get_total_in_range(username, created_after, created_before)
indent = " " * depth
lo = created_after if created_after is not None else "start"
hi = created_before if created_before is not None else "now"
print(f"{indent}Range [{lo} .. {hi}]: {total} comments")
if total == 0:
return
max_reachable = ALGOLIA_DEEP_PAGE_LIMIT # page*hitsPerPage cap
if total <= max_reachable:
# safe to page through directly
page = 0
while True:
data = first_page if page == 0 else fetch_page(username, page, created_after, created_before)
hits = data["hits"]
if not hits:
break
new = 0
for h in hits:
if h["objectID"] not in seen_ids:
seen_ids.add(h["objectID"])
all_records.append({
"id": h["objectID"],
"author": h.get("author"),
"created_at": h.get("created_at"),
"created_at_i": h.get("created_at_i"),
"comment_text": h.get("comment_text"),
"story_id": h.get("story_id"),
"story_title": h.get("story_title"),
"story_url": h.get("story_url"),
"parent_id": h.get("parent_id"),
"points": h.get("points"),
})
new += 1
print(f"{indent} page {page}: +{new} (running total {len(all_records)})")
if (page + 1) * HITS_PER_PAGE >= data["nbHits"]:
break
page += 1
time.sleep(0.5) # polite pacing, Be a nice human
else:
# bisect on time: find midpoint of range and recurse both halves
lo_i = created_after if created_after is not None else 0
# use "now" as a safe upper bound if open-ended
hi_i = created_before if created_before is not None else int(time.time())
mid = (lo_i + hi_i) // 2
print(f"{indent} -> exceeds {max_reachable}, bisecting at {mid}")
dump_range(username, lo_i, mid, seen_ids, all_records, depth + 1)
dump_range(username, mid, hi_i, seen_ids, all_records, depth + 1)
In case you don't receive an answer, an alternative to whatever they might have said or not have said, is stupidly simple: Just force yourself to write. Sit down. Lay down, stand on your head. Just do what ever you need to do to force yourself to write. Small personal essays are my suggestion. And a tip: never pause to edit -- just keep writing until you tire, and only then, if you must, go back and scrutinize yourself.
What I was asking was for anyone, prolific at writing or or not, to understand their own output in relation to…L1 cache or a context window.
But thanks! I guess my issue with writing, as a complete amateur, is the later stages, and followup, compiling or reviewing. Reading is not the only reason to write, but it is an important one.
I think it is a relevant human-scale question, relating to and understanding a paltry personal megabyte (or less) on a forum where gigabyte models are half the conversation. Its the wrong metric in a way like LoC, to compare to your comment.
If I am understanding correctly, you are also putting into perspective the dizzying contrast of human vs LLM, where the mere context window far exceeds an average lifetime of writing, which if so, yeah, ... interesting times. Before they ruined Claude with 4.7 and 5, I had more than a few extensive, hundred-page discussions which still kind of blow my mind in review. I miss it as I do some old departed friends. It would get quite amnesiac toward the ends, but still beat the hell out of many conversations I've had with PhDs.
On that note, my generated content, or product of my LLM interactions, exceeds 6GB, probably a lot more. Of course, on average, the LLM is producing far more output. But that's ~ two years of heavy, mostly research-based dialog, much of it adversarial, probing the frontier systems themselves.
Exactly right—interesting times. I’m curious whether you could estimate what percentage of the 6Gb sessions were your own writing? Of course you guide the ideas, but this word count is a different question from “how many words have you read?” (or generated, or conpiled from source :)
Yes i do, i like to put down my thoughts in writing as well as planning out strategies and such as i found out that i could not find a pen when i needed it much which was a dilemma of mine and so i started just using my laptop and phone to jot down what came to mind and was worth recording
How much more than 1Mb, do you know?
50 years of one 50 word grocery list per week (if you keep them) is 130,000 words, but not everyone writes that much, or that consistently, or for that long, or keeps complete digital copies.
Some here probably have more than 1mb in comments. There's probably a way to calculate that through the API if someone were so inclined.
I just pinged the api to dump my 10 year comment history. Yes, that definitely can be done, in a few seconds and a script.
Edit: the csv = 698kb and the json = 866kb Edit 2: api was the wrong term. I used algolia.
Great datapoint, thanks.
My “Ask HN” was not intended to focus on HN comment writing only, but since it headed that way on its own, can you share a scriptlet so that others may do more than speculate?
You ok with python?
eth0up your last comment went auto-[dead] probably because of the python. I vouched for it, think that much worked.
I would just grab it and add to a text file URL but python cares about spaces and the script is not consistently indented 4 spaces to make it into HN code. Do you by chance have a git repo you can save it to?
Another option is to sftp it to hn@nochan.net (no password), put in /pub/ and I can turn it into a URL. If you can reach it.
If you end up truly at a dead end, let me know and I'll go the sftp route. But if you would, please try dropping the rough paste into GPT, Claude, Gemini, etc, and just ask it to clean it up. I don't think there are any missing lines. If that fails let me know, but I generally avoid direct IP contact with strangers, and I am kind of not crazy about git and most things.
It's ok, I was just trying to help cwmoore get the script but it looks like they already play with LLM's so I will let them figure it out from what you pasted. Thanks for contributing your script.
I am
#!/usr/bin/env python3
""" WARNING: For use on mortals only -- WIll break on tptacek and other mutant laureates """
import csv import json import sys import time from pathlib import Pathimport requests
API = "https://hn.algolia.com/api/v1/search_by_date" HITS_PER_PAGE = 1000 ALGOLIA_DEEP_PAGE_LIMIT = 10000 # page * hitsPerPage must remain <= this
def fetch_page(username, page, created_after=None, created_before=None): params = { "tags": f"comment,author_{username}", "hitsPerPage": HITS_PER_PAGE, "page": page, } filters = [] if created_after is not None: filters.append(f"created_at_i>{created_after}") if created_before is not None: filters.append(f"created_at_i<{created_before}") if filters: params["numericFilters"] = ",".join(filters)
def get_total_in_range(username, created_after=None, created_before=None): data = fetch_page(username, 0, created_after, created_before) return data["nbHits"], datadef dump_range(username, created_after, created_before, seen_ids, all_records, depth=0): """Recursively bisect time ranges to stay under Algolia's 10k page wall.""" total, first_page = get_total_in_range(username, created_after, created_before) indent = " " * depth lo = created_after if created_after is not None else "start" hi = created_before if created_before is not None else "now" print(f"{indent}Range [{lo} .. {hi}]: {total} comments")
def main(): ap = argparse.ArgumentParser() ap.add_argument("username") args = ap.parse_args() username = args.username if __name__ == "__main__": main()It doesn't take long to build up that many words on online forums, like Hacker News.
The text content of my iOS Notes, when zipped, is about 500Kb. Some small fraction of that was copy-pasted from someone else’s writing.
I probably have an order of magnitude more than that.
Bigger than a context window, smaller than a useful model, and fits in L2 cache.
Thanks for your input. Apparently it does! There’s another commenter on this thread who found less than that in their 10 year history.
Far, far more than 1 MB.
Thanks, that’s a lot. Can I ask for a ballpark figure? And how you have written so much?
In case you don't receive an answer, an alternative to whatever they might have said or not have said, is stupidly simple: Just force yourself to write. Sit down. Lay down, stand on your head. Just do what ever you need to do to force yourself to write. Small personal essays are my suggestion. And a tip: never pause to edit -- just keep writing until you tire, and only then, if you must, go back and scrutinize yourself.
Hah! Good advice!
What I was asking was for anyone, prolific at writing or or not, to understand their own output in relation to…L1 cache or a context window.
But thanks! I guess my issue with writing, as a complete amateur, is the later stages, and followup, compiling or reviewing. Reading is not the only reason to write, but it is an important one.
I think it is a relevant human-scale question, relating to and understanding a paltry personal megabyte (or less) on a forum where gigabyte models are half the conversation. Its the wrong metric in a way like LoC, to compare to your comment.
If I am understanding correctly, you are also putting into perspective the dizzying contrast of human vs LLM, where the mere context window far exceeds an average lifetime of writing, which if so, yeah, ... interesting times. Before they ruined Claude with 4.7 and 5, I had more than a few extensive, hundred-page discussions which still kind of blow my mind in review. I miss it as I do some old departed friends. It would get quite amnesiac toward the ends, but still beat the hell out of many conversations I've had with PhDs.
On that note, my generated content, or product of my LLM interactions, exceeds 6GB, probably a lot more. Of course, on average, the LLM is producing far more output. But that's ~ two years of heavy, mostly research-based dialog, much of it adversarial, probing the frontier systems themselves.
Exactly right—interesting times. I’m curious whether you could estimate what percentage of the 6Gb sessions were your own writing? Of course you guide the ideas, but this word count is a different question from “how many words have you read?” (or generated, or conpiled from source :)
Way way more. I write about that in journal entries alone per year
Thanks for the feedback! Sounds like you are prolific and consistent!
Do you write outside of journaling? Same amount?