Convert Pandas dataframe to format suitable for PostgreSQL COPY ... FROM STDIN
NickName:Michal Charemza Ask DateTime:2022-07-20T16:07:21

Convert Pandas dataframe to format suitable for PostgreSQL COPY ... FROM STDIN

Given a Pandas dataframe, how can it be converted to a form suitable for PostgreSQL's COPY FROM STDIN (ideally in TEXT mode)?

A naive way would be:

text = df.to_csv(
    index=False,
    header=False,
    sep='\t',
    na_rep=r'\N',
)

but this won't work in several cases. For example, when there is a tab in the data itself - it wraps such fields in quotes instead of putting a backslash in front of it, which is what's required according to https://www.postgresql.org/docs/current/sql-copy.html#id-1.9.3.55.9.2

Note the data can have NAs, which should be converted to NULLs, and it can have tabs, quotes, newlines etc, which should be preserved.

Copyright Notice:Content Author:「Michal Charemza」,Reproduced under the CC 4.0 BY-SA copyright license with a link to the original source and this disclaimer.
Link to original article:https://stackoverflow.com/questions/73048026/convert-pandas-dataframe-to-format-suitable-for-postgresql-copy-from-stdin

More about “Convert Pandas dataframe to format suitable for PostgreSQL COPY ... FROM STDIN” related questions

Convert Pandas dataframe to format suitable for PostgreSQL COPY ... FROM STDIN

Given a Pandas dataframe, how can it be converted to a form suitable for PostgreSQL's COPY FROM STDIN (ideally in TEXT mode)? A naive way would be: text = df.to_csv( index=False, header=Fal...

Show Detail

Using PostgreSQL COPY FROM STDIN

Is it possible to use PostgreSQL's COPY FROM STDIN statement to load data from CSV file by passing some sort of Reader or Writer object the same way it's done in Java? What library should I use? Ko...

Show Detail

Convert JSON format to pandas dataframe source postgresql

I want to convert a json file from a postgresql source into a pandas dataframe import psycopg2 import pandas as pd # Construct connection string conn_string = ("CONNECTION_STRING") conn =

Show Detail

SQL: convert backup file from copy format to insert format

I have a PostgreSQL backup made with PHPPgadmin using Export > Copy (instead Copy > SQL which is actually what I need). File contains entries like this: COPY tablename(id, field) FROM stdin; ... ...

Show Detail

PostgreSQL copy command using STDIN

I need to load the data from the CSV file to table in PostgreSQL. and I'm not a superuser to use the copy command. when i read few topics from the postgreSQL site I came to know abut the \copy comm...

Show Detail

COPY to POSTGRESQL from STDIN in Python / Django (on Heroku)

I'm setting up a Django app on Heroku and need to convert from an sqlite3 db backend to postgresql. Unfortunately, I'm using a shared database and so cannot get direct access to the db shell with ...

Show Detail

How to upsert pandas DataFrame to PostgreSQL table?

I've scraped some data from web sources and stored it all in a pandas DataFrame. Now, in order harness the powerful db tools afforded by SQLAlchemy, I want to convert said DataFrame into a Table() ...

Show Detail

How to upsert pandas DataFrame to PostgreSQL table?

I've scraped some data from web sources and stored it all in a pandas DataFrame. Now, in order harness the powerful db tools afforded by SQLAlchemy, I want to convert said DataFrame into a Table() ...

Show Detail

How to convert pandas DataFrame to dictionary for Newick format

I have the following dataset: import pandas as pd df = pd.DataFrame([['root', 'b', 'a', 'leaf1'], ['root', 'b', 'a', 'leaf2'],

Show Detail

Pandas data from stdin

Is it possible to have stdin data go into a pandas DataFrame? Currently I'm saving the data in an intermediate json file and then doing: pandas.read_json('my_json_file.json') but was wondering i...

Show Detail