Data Import and Export

The civis.io namespace provides several functions for moving data in and out of Civis.

Tables

Often, your data will be in structured format like a table in a relational database, a CSV, or a dataframe. The following functions handle moving structured data to and from Civis. When using these functions, it is recommended to have pandas installed and to pass use_pandas=True in the appropriate functions. If pandas is not installed, data returned from Civis will all be treated as strings.

civis_to_csv(filename, sql, database[, ...])

Export data from Civis to a local CSV file.

civis_to_multifile_csv(sql, database[, ...])

Unload the result of SQL query and return presigned urls.

civis_file_to_table(file_id, database, table)

Upload the contents of one or more Civis files to a Civis table.

csv_to_civis(filename, database, table[, ...])

Upload the contents of a local CSV file to Civis.

dataframe_to_civis(df, database, table[, ...])

Upload a pandas dataframe into a Civis table.

read_civis(table, database[, columns, ...])

Read data from a Civis table.

read_civis_sql(sql, database[, use_pandas, ...])

Read data from Civis using a custom SQL string.

export_to_civis_file(sql, database[, ...])

Store results of a query to a Civis file

split_schema_tablename(table)

Split a Redshift 'schema.tablename' string

Files

These functions will pass flat files to and from Civis. This is useful if you have data stored in binary or JSON format. Any type of file can be stored in platform via the files endpoint.

civis_to_file(file_id, buf[, client])

Download a file from Civis.

dataframe_to_file(df[, name, expires_at, client])

Store a DataFrame as a CSV in Civis Platform

file_id_from_run_output(name, job_id, run_id)

Find the file ID of a File run output with the name "name"

file_to_civis(buf[, name, client])

Upload a file to Civis.

file_to_dataframe(file_id[, compression, client])

Load a DataFrame from a CSV stored in a Civis File

file_to_json(file_id[, client])

Restore JSON stored in a Civis File

json_to_file(obj[, name, expires_at, client])

Store a JSON-serializable object in a Civis File

Databases

These functions move data from one database to another and expose an interface to run SQL in the database. Use query_civis() when you need to execute SQL that does not return data (for example, a GRANT or DROP TABLE statement).

transfer_table(source_db, dest_db, ...[, ...])

Transfer a table from one location to another.

query_civis(sql, database[, client, ...])

Execute a SQL statement as a Civis query.