IndexedDB and MongoDB - local and remote updates

December 12, 2016 - reworked to the actual system state

The browser-processing uses ajax, node.js and MongoDB for data access and data storage.

The app uses:

• local data access and data storage based on IndexedDB
• additional remote data access and data storage based on ajax, node.js and MongoDB

Synchronization is done as follows:

• in the browser there is no synchronization necessary
• in the app you have
• calendar: comparison of the dates with data in the local IndexedDB and the remote MongoDB when the calendar is shown - indicated by the color of the days in the calendar
• day entry: when the user chooses a day to enter more data or just see the data, the system checks if there is a local record and a remote record, if both are there, a synchronization for the user and day is done
• this synchronization at moment is only put to the server, if the user changes the record oder enters more data
• mass-synchronization: the user has to choose this menue point, the system checks
• for the last 100 days for a user
• if a record is local and not remote, then it is transferred from local to remote
• if a record is remote and not local, then it is transferred from remote to local
• a comparison and synchronization in case a day has local and remote data is not done, that is only done when day-data are entered (see above)

If a user has a clear strategy to update his data only with the app, then nothing bad can happen, the system synchronizes when new data is entered and in case the remote system is not available the mass-synchronization can be activated - or the days are invoked one by on depending on the color in the calendar

The diagnosis of problems, when requests to read to change data go to the server will be handled in an expanded way:

• try-catch for connectivity breakdown (get's added)
• ajax-fail for problems within the connectivity, like false request path or false response data
• ajax-done with no data as "special" result

The solution does not only require new code, but also new principles for the appMode:

• getting data
• if a server connection is available, read from server connection (AJAX, MongoDB) first
• if appMode, read locally from IndexedDB 
• check if the corresponding local data in IndexedDB has to be synchronized
• always use logical keys and not technical keys (username and date and not UID's of any kind)
• use hashcodes to determine, if synchronization is necessary
• do the synchronization
• if synchronization has been done and a server connection is available, write synchronized data to sever (AJAX, MongoDB)
• if synchronization has beed done, write synchronized data to local indexedDB
• put synchronized data to the ui
• this way a smooth synchronization is done automatically and incrementally, transparent for the user
• putting data

• at first always put data locally (insert, update with merge) to IndexedDB
• then put data remote, if a server connection is available (AJAX, MongoDB)
• the remote server does synchronize and merge for updates
• synchronization is done automatically in the background
• the app sets a flag if the server connection is not available
• a loop checks it the server connection is available again 
• "keep alive" check
• long waiting time, progressive from 1 minute to 20 minutes after 2 hours
• if the server is available again then the synchronization is started automatically
• the user just gets a message and a flag in the footer region to indicate the synchronization

Not all functions and data have to be supported locally. The critical functions for local and remote support are:

• initDB - initializing functions for the database, the document store and indexed
• login - a function that checks the userdata, locally that will be done against a cookie, but the login protocol data is stored in the database
• putTagesdaten - a function to save the diary-data that has been entered
• depending on the data it's insert or update respectively "upsert"
• regarding the data content on the server a merge logic with aggregation of income, expenses and workhours is necessary, the app is single user in local mode
• getCalendar - a function retrieving the dates, for which diary data is in the database, the dates are compared to the dates in IndexedDB - this can be used to synchronize
• getTageListe - a function which retrieves detail data with certain selection logic
• getTageListe - for a special date and username
• getTageListe - for the day with data before a selection-date
• getTageListe - for the day with data after a selection-date
• getTageListe - for all data of the username 
• the function for coreading will only be available on the browser
• putUserData - will be a new function for users to change their password and their eMail address
• getUserData - will be a new function for users to get their eMail address (the password is not transmitted from the server, but it's checked in the server)

Technically the code from node.js will be used as basis for the corresponding functions in the app.

The integration follows clear principles:

• the calls to the existing functions on the server are already there as ajax invocations
• these calls will be wrapped by async.waterfall, if that is not already the case
• the async.waterfall gets new functions for the local data access and data storage
• the functions with the ajax-calls get wrapped with conditions regarding server availability

The it's not a very big deal. The synchronization has already been programmed as manually started batch processing - this will be changed by automatic invocation after connection is reestablished after it was missing for a while. Classically it's a watchdog-mechanism.

Is smooth synchronization enough?
Smooth synchronization means that data is synchronized on demand of requests to the server. Two aspects:

• when the calendar control is shown it can be supplied by the server or by IndexedDB - not with synchronization 
• when a record is read it is read from server and from indexedDB - not yet with synchronization

What will be the best strategy?

• calendar display can mark which records are not in indexedDB, they can be colored differently
• automatic synchronization on calendar display can take some time, that will disturb the user
• the user can click on dates, that require synchronization and that does the synchronization, even it no input was provided (meaning the dirty flag is set automatically)
• but that makes synchronization a probabilistic matter
• if a record is worked on, then it is synchronized in any case

A new synchronization logic is implemented, based on username and isoday.

• Calendar-based synchronization
• the getCalendar-API to the server gives extended information with a hashcode
• the hashcode is calculated for indexedDB
• comparision
• action
• the calendar control has new coloring
• blue - data only on server
• red - data only local
• green - data is on server and local
• priority is given to save data from local to the server
• if red then the update to the server is called immediately, that's done in the thinktime of the user, at least it should be
• Diary-entry based synchronization
• hashcode-check is done when server is available by comparing, if different hashcodes, then an update to the server is done, the server merges and the reading of the server is repeated.
• when the user updates, both systems get the latest version

Decision:

• the getCalendar mechanism which compares the dates with data in IndexedDB and in the server will show a button for synchronization and the user can choose to do it
• the synchronization will be done with the existing system - checking picture-synchronization will be done

ntasyncpage wird genutzt