HTTP file system extension

The httpfs extension extends the Kuzu file system by allowing reading from or writing to files hosted on remote file systems. The httpfs extension provides support for the following remote file systems:

• This extension: Plain HTTP(S) file systems
• S3 extension: Object storage via the AWS S3 API
• GCS extension: Object storage via the Google Cloud Storage (GCS) API

If you’re looking to use Kuzu with Azure Blob Storage, see the Azure extension.

Note

The S3 and GCS extensions are implemented as features of the httpfs extension.

• Over plain HTTP(S), Kuzu only supports reading files.
• For object storage via the S3 or GCS API, Kuzu supports reading, writing, and globbing files.

Usage

INSTALL httpfs;
LOAD httpfs;

HTTP(S) file system

Using the httpfs extension, you can read from a file hosted on an HTTP(S) server in the same way as you would from a local file. The example below shows how to scan data from a CSV file hosted on an HTTP(S) server.

LOAD FROM "https://extension.kuzudb.com/dataset/test/city.csv"
RETURN *;

Waterloo|150000
Kitchener|200000
Guelph|75000

Local cache

Remote file system calls can be expensive and highly dependent on your network conditions (bandwidth, latency). Queries involving a large number of file operations (read, write, glob) can be slow. To expedite such queries, we introduce a new option: HTTP_CACHE_FILE. A local file cache is initialized when Kuzu requests the file for the first time. Subsequent remote file operations will be translated as local file operation on the cache file.

The example below shows how to enable the local cache for remote files.

CALL HTTP_CACHE_FILE=TRUE;

Tip

Cached files are visible per transaction. Therefore, if you have set HTTP_CACHE_FILE=TRUE and then run a LOAD FROM statement on a remote file, say LOAD FROM "https://example.com/city.csv" RETURN *;, then this file will be downloaded first and then scanned locally from the downloaded file. If you run the same LOAD FROM statement again, it will be downloaded again from the remote URL. This is because the second statement is executed as a separate transaction and we do not know if the already downloaded remote file has changed since the last time Kuzu downloaded it.

If you need to scan the same remote file multiple times and benefit from caching across multiple scans, you can run all the LOAD FROM statements in the same transaction. The example below shows how to do this.

BEGIN TRANSACTION;
LOAD FROM "https://example.com/city.csv" RETURN *;
LOAD FROM "https://example.com/city.csv" RETURN *;
COMMIT;

Now the second LOAD FROM statement will run much faster because the file is already downloaded and cached and the second scan is within the same transaction as the first one.