Mitch Pronschinske is the Lead Research Analyst at DZone. Researching and compiling content for DZone's research guides is his primary job. He likes to make his own ringtones, watches cartoons/anime, enjoys card and board games, and plays the accordion. Mitch is a DZone Zone Leader and has posted 2576 posts at DZone. You can read more from them at their website. View Full User Profile

Datastore Controls in the 1.3.2 GAE SDK

03.30.2010
| 7700 views |
  • submit to reddit
The GAE team delivered the 1.3.2 SDK update last week, and with it came significant new datastore controls.  The SDK allows GAE developers to use eventually consistent reads and datastore deadlines.  The SDK also includes API enhancements and new security features.

Specifying Eventually Consistent Reads
GAE now provides an option to specify eventually consistent reads for fetches and datastore queries.  The eventually consistent read policy works on a per-call basis, allowing the datastore to read data from a secondary storage location when the primary location is unavailable.  The secondary location should be similar to the primary location's data copy even though it may not have all the changes from the primary location when the data is read.  

The eventually consistent read policy is useful because you can sacrifice some consistency for higher availability.  The constant updating of the strong consistency read policy is not necessary in many cases.  The strong constancy read policy is the default method for GAE datastore updates and fetches, which takes data from the primary storage location only.  When a node at the primary location goes down or becomes unavailable, the read waits for the node to return.  This feature in the strongly consistent read policy will sometimes cause the read to miss your request handler deadline.  

With eventually consistent reads, you tend to see a reduction in datastore time-outs and error responses during queries and fetches.  The recent update to the GAE SDK still keeps strong consistency as the default, but now developers have the option of using eventual consistency, which the GAE developers recommend you use liberally.

Specifying Datastore Deadlines
The GAE datastore added another important feature with the 1.3.2 SDK.  Users can now specify deadlines for datastore calls.  If the call is not completed before the deadline, the call is aborted so that the application execution is not completely interrupted.  The GAE datastore automatically retries calls for up to 30 seconds, but now users can set a deadline that is less than 30 seconds for applications that are more latency-sensitive.  Applications can also do other things when a request takes too long, like cache referencing or displaying less data.  This gives your application more control over its performance.   

In Java, you can enable deadlines and set an eventually consistent read policy by calling the methods addExtension() and setTimeoutMillis() respectively to a query object:
Query q = pm.newQuery(Employee.class);
q.addExtension("datanucleus.appengine.datastoreReadConsistency", "EVENTUAL");
q.setTimeoutMillis(2000);
Deadlines and eventual consistency can be used in JDO and JPA with configuration, or with the low-level Java datastore API.  

For Python, you need to create an RPC object.  Use the create_rpc() function and set a deadline and the read_policy on the object.  Finally, you pass the RPC object to the call as an argument.  This is an example using a datastore fetch:
rpc = db.create_rpc(deadline=7, read_policy=db.EVENTUAL_CONSISTENCY)
results = Employee.all().fetch(10, rpc=rpc)
Along with new datastore controls, there were several other useful new features in the 1.3.2 SDK release.  A new Blobstore method allows your application to request a blob's content from within the application code.  The SDK also expands the number of ports you can access with the URLFetch API and the mail attachments types for the Mail API.  For improved security, a new Denial of Service blocking system has been added.  Finally, there is a new Java version of the Appstats profiling tool.