Data Import and Export

The civis.io namespace provides several functions for moving data in and out of Civis.

Tables

Often, your data will be in structured format like a table in a relational database, a CSV, or a dataframe. The following functions handle moving structured data to and from Civis. When using these functions, it is recommended to have pandas or polars installed and to pass return_as="pandas" or return_as="polars", respectively, in the appropriate functions. If pandas or polars is not installed, data returned from Civis will all be treated as strings.

civis_to_csv(filename, sql, database[, ...])	Export data from Civis to a local CSV file.
civis_to_multifile_csv(sql, database[, ...])	Unload the result of SQL query and return presigned urls.
civis_file_to_table(file_id, database, table)	Upload the contents of one or more Civis files to a Civis table.
csv_to_civis(filename, database, table[, ...])	Upload the contents of a local CSV file to Civis.
dataframe_to_civis(df, database, table[, ...])	Upload a dataframe into a Civis table.
read_civis(table, database[, columns, ...])	Read data from a Civis table.
read_civis_sql(sql, database[, return_as, ...])	Read data from Civis using a custom SQL string.
export_to_civis_file(sql, database[, ...])	Store results of a query to a Civis file
split_schema_tablename(table)	Split a Redshift 'schema.tablename' string

Files

These functions will pass flat files to and from Civis. This is useful if you have data stored in binary or JSON format. Any type of file can be stored in platform via the files endpoint.

civis_to_file(file_id, buf[, client])	Download a file from Civis.
dataframe_to_file(df[, name, expires_at, ...])	Store a dataframe as a CSV in Civis Platform.
file_id_from_run_output(name, job_id, run_id)	Find the file ID of a File run output with the name "name"
file_to_civis(buf[, name, expires_at, ...])	Upload a file to Civis.
file_to_dataframe(file_id[, return_as, ...])	Load a dataframe from a CSV stored in a Civis File.
file_to_json(file_id[, client])	Restore JSON stored in a Civis File
json_to_file(obj[, name, expires_at, ...])	Store a JSON-serializable object in a Civis File

Databases

These functions move data from one database to another and expose an interface to run SQL in the database. Use query_civis() when you need to execute SQL that does not return data (for example, a GRANT or DROP TABLE statement).

transfer_table(source_db, dest_db, ...[, ...])	Transfer a table from one location to another.
query_civis(sql, database[, client, ...])	Execute a SQL statement as a Civis query.