jconline543
No user record in our sample, but jconline543 has activity below (stories or comments). Likely we have partial data — the full bulk-load will fill profiles in.
No user record in our sample, but jconline543 has activity below (stories or comments). Likely we have partial data — the full bulk-load will fill profiles in.
Also, for querying both recent and historical data together, you wouldn't need to modify this tool at all. You could just add a separate periodic job (e.g. hourly/daily) that copies recent data to S3: sqlCopyCOPY…
Thanks for your interest! Let me address your questions: Querying the data: Yes, you can easily query the Parquet files with DuckDB. The files are stored in a year/month partitioned structure (e.g.,…
If you just need the Postgres -> S3 archival pattern you described, I built a simpler focused tool: pg-archiver (https://github.com/johnonline35/pg-archiver) It: - Auto-archives old Postgres data to Parquet files on S3…