[{"data":1,"prerenderedAt":4},["ShallowReactive",2],{"post-content-query-gigabyte-logs-with-duckdb":3},"\u003Cp>Last week a teammate sent me a 1.2 GB CSV export and asked which API keys had tripped the rate limiter more than fifty times in the past month. The file came from a dashboard nobody in the room remembered setting up. Classic.\u003C\u002Fp>\n\n\u003Cp>My old instinct would have been to reach for MySQL. Spin up a container, write a CREATE TABLE, fight LOAD DATA INFILE until the column types stopped guessing wrong, and finally run the actual query. The import script alone would have outlived the analysis. These days I open DuckDB first, not a MySQL container.\u003C\u002Fp>\n\n\u003Cp>I didn't do any of that. One command, eleven seconds, answer on the screen.\u003C\u002Fp>\n\n\u003Cpre>\u003Ccode class=\"language-sql\">SELECT api_key, count(*) AS hits\nFROM read_csv('export.csv')\nWHERE status = 429\nGROUP BY api_key\nHAVING count(*) > 50\nORDER BY hits DESC;\u003C\u002Fcode>\u003C\u002Fpre>\n\n\u003Cp>That's DuckDB, and you've probably seen it on Hacker News this week since the company behind it just got bought by AWS. I'll leave the billion-dollar M&A takes to other people. Here's what I actually do with it on a normal Tuesday: it's the embedded database that treats your files as tables. There's no server to start and no import step to babysit. Point SQL at a CSV, a JSON file, or a Parquet file and it answers.\u003C\u002Fp>\n\n\u003Ch2>Why the import pipeline was always the wrong step\u003C\u002Fh2>\n\n\u003Cp>MySQL is a database you run. It's great at that. For a one-off question about a file someone emailed you, the entire ceremony is dead weight: schema, migration, import script, and the container you forget to stop until your laptop melts.\u003C\u002Fp>\n\n\u003Cp>DuckDB is the opposite. It's an in-process library, not a service, so there's nothing to provision and nothing to clean up. The schema is inferred from the data when you query, which means a CSV that grows a new column on line three million doesn't break anything. And it's columnar. When I write SELECT count(*) FROM 'logs.parquet', DuckDB reads the row group metadata and only the columns it needs. It doesn't open the whole file like a kid flipping through every page to find the one with the dog in it.\u003C\u002Fp>\n\n\u003Cimg src=\"https:\u002F\u002Fimages.unsplash.com\u002Fphoto-1555066931-4365d14bab8c?w=800&q=80\" alt=\"Developer writing SQL queries against log files in a dark terminal\" width=\"800\" loading=\"lazy\" style=\"width:100%;border-radius:12px;margin:1.5em 0;\" \u002F>\n\n\u003Cp>There's a real speed story in there, but let me be honest about the trade-offs too, because I keep seeing people oversell this thing.\u003C\u002Fp>\n\n\u003Cp>DuckDB is an OLAP engine. It will happily scan hundreds of gigabytes while your fan sounds like a leaf blower, and when a file is truly enormous it spills to disk instead of dying. What it is not is a transaction store. I would not back an app with it. SQLite and MySQL still own that job, and I reach for DuckDB several times a week anyway. Different tools, different corners of my desk.\u003C\u002Fp>\n\n\u003Ch2>The one-liners I actually use\u003C\u002Fh2>\n\n\u003Cp>JSON Lines event logs are a favorite. DuckDB works out the nested fields without being told:\u003C\u002Fp>\n\n\u003Cpre>\u003Ccode class=\"language-sql\">SELECT event, count(*) AS n\nFROM read_json_auto('events.jsonl')\nGROUP BY event\nORDER BY n DESC;\u003C\u002Fcode>\u003C\u002Fpre>\n\n\u003Cp>Everything in a folder becomes one table with a glob, which is how I get through a month of logs without writing a loop:\u003C\u002Fp>\n\n\u003Cpre>\u003Ccode class=\"language-sql\">SELECT * FROM 'logs\u002F2026-08-*.parquet';\u003C\u002Fcode>\u003C\u002Fpre>\n\n\u003Cp>And the CLI is good enough that half my analysis never touches Python. I keep a shell alias shaped like this one because I type it a dozen times during a bad week:\u003C\u002Fp>\n\n\u003Cpre>\u003Ccode class=\"language-bash\">duckdb -c \"SELECT status, count(*) AS n FROM read_csv('logs\u002F2026-08.csv') GROUP BY status ORDER BY n DESC\"\u003C\u002Fcode>\u003C\u002Fpre>\n\n\u003Cp>The one that still gets me is remote files. Install the httpfs extension and DuckDB will query a Parquet file on S3 or a public URL without downloading it first:\u003C\u002Fp>\n\n\u003Cpre>\u003Ccode class=\"language-sql\">INSTALL httpfs;\nLOAD httpfs;\n\nSELECT * FROM 'https:\u002F\u002Fexample.com\u002Fdata.parquet';\u003C\u002Fcode>\u003C\u002Fpre>\n\n\u003Cp>For a solo developer that borders on magic. I've analyzed public datasets this way and never once felt the urge to set up a warehouse.\u003C\u002Fp>\n\n\u003Ch2>Python, but only when I need it\u003C\u002Fh2>\n\n\u003Cp>When the answer needs to become a chart or feed a script, the Python binding hands you a DataFrame directly:\u003C\u002Fp>\n\n\u003Cpre>\u003Ccode class=\"language-python\">import duckdb\n\ndf = duckdb.sql(\n    \"SELECT * FROM 'events.parquet' WHERE user_id = 42\"\n).df()\u003C\u002Fcode>\u003C\u002Fpre>\n\n\u003Cp>The SQL does the heavy lifting, so pandas only ever sees the small answer. I don't get a two gigabyte DataFrame materializing in memory, and I can stop pretending I'll read the columns I need later.\u003C\u002Fp>\n\n\u003Ch2>Where I draw the line\u003C\u002Fh2>\n\n\u003Cp>I keep one persistent file, data.duckdb, for questions I ask more than once. It starts the same REPL but remembers tables between sessions. Everything else follows a rule I stole from a smarter person: the query is the script. The file is disposable, the SQL is not.\u003C\u002Fp>\n\n\u003Cp>That's why the two or three DuckDB one-liners I reuse live in \u003Ca href=\"\u002Fsnippetark\u002F\">Snippet Ark\u003C\u002Fa> now instead of a terminal history nobody scrolls. The rate limit query gets asked every month, and six months from now I'll still know why it exists. That's more than I can say for most of the imported tables in my MySQL career.\u003C\u002Fp>\n",1787912413009]