How Do You Find the Largest Files in a Data Download?
From the earlier reference collection. Read the current software blog.
Sort a directory listing by size to identify which downloaded files dominate storage, transfer, or processing time.
Measure before guessing
A large download may contain a few heavyweight tables alongside many tiny metadata files. Listing every filename alphabetically does not show which ones are worth investigating first. On systems with GNU core utilities, du -ah download/ | sort -h | tail -n 20 reports the largest directory entries in human-readable units. The -a includes files as well as directories, while sort -h understands suffixes such as K, M, and G.
For a flat directory, ls -lhS download/ sorts files by size, largest first. That can be a convenient quick look, though ls output is intended for people and can be awkward to parse in scripts. If nested folders matter, du is usually more informative because it reports the space consumed by both files and directories.
Separate file sizes from directory totals
A directory’s reported total includes its contents. If a download has a 900 MB folder and a 700 MB file within that folder, ranking all du lines can make the parent folder look like a separate giant object. Use du -ah to locate individual files, then inspect the entries carefully; use du -sh download/* when you want a summary by immediate child.
For example, suppose the listing shows stop_times.txt at 1.4 GB and routes.txt at 12 KB. That points to the stop-time table as a likely source of long parsing or transfer time, but it does not tell you how many useful records it contains. Compression can also change the picture: a small ZIP may expand to a much larger collection of text files.
Keep units and scope clear
Check whether the tool is reporting apparent file length or disk space actually allocated. Sparse files, compression, and filesystem block sizes can make those values differ. For a simple download directory on a typical filesystem, du is a useful operational estimate; stat can show file metadata when exact byte length is what you need.
Be explicit about the directory being measured. A command run from the wrong working directory may rank an unrelated cache or include old downloads. A stable pattern is to provide the path directly and redirect output to a report only after confirming the result looks right.
Use the result to guide the next step
Largest-file reports help you choose where to profile, what to transfer selectively, or which inputs might benefit from streaming rather than loading wholly into memory. They do not prove that the largest file is inefficient or disposable. A large table may be essential, while a small index may control the meaning of every record.
For repeatable comparisons, save the command, directory, and date with the output. If two downloads differ substantially in their largest entries, inspect their manifests or checksums before attributing the change to a data update. The size listing narrows the investigation; content checks explain the cause.