Skip to content

Instantly share code, notes, and snippets.

@bzerangue

bzerangue/json-to-ndjson.md

Last active April 15, 2026 11:39
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save bzerangue/7bf6610079659e57b8d50ecb94928c31 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save bzerangue/7bf6610079659e57b8d50ecb94928c31 to your computer and use it in GitHub Desktop.
Download ZIP
JSON to NDJSON
Raw
json-to-ndjson.md

NDJSON is a convenient format for storing or streaming structured data that may be processed one record at a time.

  • Each line is a valid JSON value
  • Line separator is ‘\n’

1. Convert JSON to NDJSON?

cat test.json | jq -c '.[]' > testNDJSON.json

With this simple line of code, you can convert and save files in NDJSON format.

Note: jq is a lightweight and flexible command-line JSON processor.
https://stedolan.github.io/jq/

Source: https://medium.com/datadriveninvestor/json-parsing-error-how-to-load-json-into-bigquery-successfully-using-ndjson-2b7d94616bcb

@draxil
Copy link
Copy Markdown

draxil commented Aug 9, 2022

jq can choke on very large files, and be slow. For these situations I made json2nd.

@bzerangue
Copy link
Copy Markdown
Author

bzerangue commented Nov 15, 2022

jq can choke on very large files, and be slow. For these situations I made json2nd.

Thanks for sharing @draxil !

@fixdollar7-cyber
Copy link
Copy Markdown

fixdollar7-cyber commented Apr 15, 2026

Very handy jq approach 👍

For people who need a quick browser-based alternative without command line setup, I’ve been using:

https://jsonviewertool.com/json-ndjson

It’s useful when validating API responses, BigQuery imports, JSONL datasets, or large array payloads before pushing them into pipelines.

The nice part is it instantly converts:

[
  {"id":1},
  {"id":2}
]

into:

{"id":1}
{"id":2}

which is perfect for NDJSON / JSONL workflows.

The jq command is still ideal for CI scripts, but the browser tool is great for debugging and quick checks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment