How to export – import data from App Engine's Datastore in java development.
Par Ibrahim Dawid le mardi 8 juin 2010, 18:01 - GWT - Lien permanent
Today I am going to talk of one of those things that can get you bordered
many hours Googling the web before find out the right way to do it.
I just land in the GWT island with Duke, on his Google App Engine, coming
from the old Java EE world. Working on Fastcall I had to export - import data
from the Google's App Engine Datastore. At first glance, I expected to do
it in two or three clicks, but that was not the case.
Very quickly I get stuck because my project was developed on the Java
runtime environment where the native support for export – import data is
missing.
So if you are concerned with that, take a closer look at this post.
1- Due to the use of python, we’ll install on our local machine:
“Google App Engine SDK for Python”
(http://code.google.com/appengine/downloads.html) and
the python interpreter from http://www.python.org/download/.
2- In our application’s web.xml deployment descriptor file we will add the
following servlet com.google.apphosting.utils.remorteapi.RemorteApiServlet in
order to interact with our web application using python commands.
------------------------------------------------------------
<servlet>
<servlet-name>remoteapi</servlet-name>
<servlet-class>com.google.apphosting.utils.remoteapi.RemoteApiServlet</servlet-class>
</servlet>
<servlet-mapping>
<servlet-name>remoteapi</servlet-name>
<url-pattern>/remote_api</url-pattern>
</servlet-mapping>
<security-constraint>
<web-resource-collection>
<web-resource-name>remoteapi</web-resource-name>
<url-pattern>/remote_api</url-pattern>
</web-resource-collection>
<auth-constraint>
<role-name>admin</role-name>
</auth-constraint>
</security-constraint>
------------------------------------------------------------
3- Deploy your web application on the App Engine.
So for so good, the first part of the work is done!!!!
Next we will see two kind of import - exports:
A- Raw data import - export
We can make use of the python commands to do some import - export
tasks:
Note: In the commands below you can ommit the –kind=<…> argument. In
that case all our entities will be backup.
Dump: (be sur that the our_dump.txt doesn’t exist)
bulkloader.py --dump --app_id=<our_app_id> --kind=<our_entity>
--url=http://<our_app_name>.appspot.com/remote_api
--filename=<our_dump.txt>
Restore:
bulkloader.py --restore --app_id=<our_app_id> --kind=<our_
entity> --filename=<our_dump.txt>
--url=http://<our_app_name>.appspot.com/remote_api
When running each of those commands, you will be prompted for
authentication. Just put your Google email account and you password.
B- Formatted data import – export
We'll use the bulkloader.yaml configuration file to describe how
your data should be transformed when uploaded or downloaded. But where is this
file??? Well, we are going to create it with this command:
appcfg.py create_bulkloader_config --filename=bulkloader.yaml --url
http://<our_app_id>.appspot.com/remote_api
Then let’s customize it.
Go to the transformers section and do your own customization: This
configuration below will probably be not suitable for you. Use it just like a
sample.
------------------------------------------------------------
transformers:
- kind: Contact
connector: CSV
property_map:
- property: __key__
external_name: email
export_transform:
transform.key_id_or_name_as_string
- property: name
external_name: name
- property: surname
external_name: prenom
- property: numTelF
external_name: numTelF
- property: numTelM
external_name: numTelM
- property: Fax
external_name: Fax
- property: adresse
external_name: adresse
------------------------------------------------------------
This file has 2 main parts:
The connector configuration: lets you choose the type of the backup file.
Available connectors are csv, xml or simpletext. The last connector can only be
used for export not import (keep that in mind).
The property mapping: this is where you define the properties you want to
export, the mappings between the entity properties and the exported names and
finally some import – export transformation functions that will occur during
the process.
The kind keyword means our persisted entity class: Contact in our
case.
Run the following commands:
Note: the –kind=<…> is mandatory this time.
Dump:
appcfg.py download_data --config_file=bulkloader.yaml
--filename=<filename> --kind=<our_ entity> --url
http://<our_app_id>.appspot.com/remote_api
Restore:
appcfg.py upload_data --config_file=bulkloader.yaml
--filename=<filename> --kind=<our_ entity> --url
http://<our_app_id>.appspot.com/remote_api
And that's it.
For any further information have a quick look at
http://code.google.com/intl/fr-FR/appengine/docs/python/tools/uploadingdata.html
