Tuesday, October 25, 2016

the diary data structures

December 16, 2016, BERICHT changed to array and each element is a text and it can have an additional array of comments
The diary data structure is a JSON-structure or javascript-object and the datastructure that is stored into indexedDB and MongoDB.

the diary data structure
{
username - string
isoday - string - day as the user feels it locally (YYYY-MM-DD)
recid -  string - Satzart
DATUM - string - Timestamp, ISO format, zero hours of day, so timeoffset is compensated
LOCDATUM - string - external format timestamp
BERICHT - so far: string, will be changed to array
  [i] with structure {}

  status – string – active or inactive
  privacy - string - level private, friends, public
  text - string - normal text or html richtext
  locts - string - local timestamp (from username-client, here diary owner)
  location - string - information to location, where the diary entry has been made
  COMARRAY[j] - array of structures with comments to the text[i] with
    comment - string
    username - string - user woh made the comment
    locts - string - local timestamp (from username-client)
    location - string - information to location, where the diary entry has been made
GESUND - int
WETTER - int
STIMMUNG - int
AUSGABEN - array with fields
 bezug - real
 betrag - real
EINNAHMEN - array with fields, like AUSGABEN
ZEIT - array with fields, like AUSGABEN
geo - string - with data regarding location
tscreated - timestamp - from the server, when it stores the data
}

the message data structure


{
            channel: channel,
            level1: level1,
            level2: level2,
            severity: severity,
            timestamp: timestamp,
            tscreated: tscreated,
            ip: ip,
            username: username

        };

Some important parameters, stored in sysInfo in uisystem Module. sysInfo is accessed via uisystem.getsysInfo().



"appMode": false, 
"server1": "http://localhost:3000", 
"server1message": "OK", 
"activeServer": "http://localhost:3000", 
"server2": "", 
"doAjax": true, 
"doIndexedDB": true, 
"myIndexedDB": {}, 
"info": "
Cookies müssen erlaubt sein
Cookie Status:true
Geo-Location sollte erlaubt sein
Die IP-Adresse wird gespeichert
Windows
Viewport Width:537
Language:de
Geolocation:Latitude: 50.782884499999994
Longitude: 6.9200744"
"cookieEnabled": true, 
"os": "Windows", 
"device": "Netscape 5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.71 Safari/537.36 Win32 (true)", 
"appName": "Netscape", 
"appVersion": "5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.71 Safari/537.36", 
"platform": "Win32", 
"viewportwidth": 537, 
"navlang": "de", 
"geolocation": "Latitude: 50.782884499999994
Longitude: 6.9200744" }

the most important parameters are:
  • appMode - true=application, false=browser with node.js backend
  • doIndexedDB - true=IndexedDB can be used, based on availability test, usually in app
  • doAjax - true=connection is available to activeServer
  • activeServer - hostname and port of the server that is used as backend for the diary
  • pushServer - hostname and port of the server 

Saturday, October 22, 2016

developers diary

To trace the ongoing testing, enhancing and enriching a chronological post is added - the diary of the diary development.
November 25, 2016
The special rights of the administrator are protected:

  • the js-Files with administrator-functions are only provided in browser mode and only when an administrator has logged in
  • the administrator menue functions are only displayed in browser mode and only when an administrator has logged in
  • the administrator api's an the ajax server are checked additionally regarding an administrator that has logged in


November 23, 2016
Big steps to the finish habe been done. But still some points to do:
  • done - text output in listviews is sometimes cut, the error is known in the internet but did not find yet the right recipe - the only chance is to inactivate complex html by escaping it or filtering the pure text
  • done - convenient change of password and email-address for normal users
  • done- check if new registered username is already in the system
  •  done- serverside check for administrator-role, not yet for all the functions
  • done - complete dynamic upload of scripts for the administrator functions and check of the upload on the server regarding the administrator role
  • done -  control of all content areas regarding scrolling and formatting with max-width
  • open - experimental grid-design with two columns for very wide screens
  • open - tests on iPhone with browser and with app, emulation and safari on Windows 10 work well
To avoid that text is cut in listviews:
var ber = record.BERICHT;
ber = ber.replace(/<br>/g, "#br%");
ber = ber.replace(/<br\/>/g, "#br%");
ber = ber.replace(/<br \/>/g, "#br%");
ber = "<span>" + ber + "</span>";
bertxt = $(ber).text();
bertxt = bertxt.replace(/#br%/g, "<br/>");

There is html in record.BERICHT und therefore the new line function is preserved, the other functions are removed because the are the reason for the text cutting. css with white-space: normal is necessary on the li level resp. the text for the a in li. For the raw display html has to be escaped:

var ber = record.BERICHT;
record.BERICHT = ber.replace(/&/g, "&amp;").replace(/</g, "&lt;").replace(/>/g, "&gt;");

   

November 20, 2016
Validation of passwords and email address with regex, looks good.
Problem: when focus is set, that does not position the field into a visible area. 
Solution 1: validate fields directly after input, then positioning is not necessary
Solution 2: I find a solution for the positioning - scrollParent does not work well, because the scrollbar is on the body resp. the page. My function scroll2Last now explicitly uses body instead of scrollParent

November 18 and 19, 2016
Concentration on beautifying the pages, learned about conflicts when different pages are changed by code, because the layout of the active page can be changed accidentally. The contents of other pages may be changed, but not the layout, that can lead to conflicts. Layout changes should be done when the target pages are to be shown.

November 17, 2016
If a new user does login in the app, the old, local data still refer to the old user. For data privacy these old data are delete automatically.
After a user changed old data are shown for some milliseconds (the well known flickering of jqm), so the event sequence for page change is checked in detail to avoid that technical effect.
http://stackoverflow.com/questions/11576602/jquery-mobile-page-events-order
and http://www.gajotres.net/page-events-order-in-jquery-mobile/ shows:
pagebeforechange
Page 2 – pagebeforecreate
Page 2 – pagecreate
Page 2 – pageinit
Page 1 – pagebeforehide
Page 2 – pagebeforeshow
Page 1 – pagehide
Page 2 – pageshow
pagechange
Maybe there is a simpler solution from http://rickluna.com/wp/2013/06/preventing-jquery-mobile-page-flicker/:
.ui-page {
  -webkit-backface-visibility: hidden;

}

November 16, 2016
Colorizing listview elements was a little bit critical, solution:
        $('#usersListLink').css({
            "background-color": "yellow",
            "text-shadow": "none"
        });
The listview has li elements with an a anchor in it, the usersListLink refers to the anchor, not the list element in li of a certain element identified by #userListLink. 
A logout function has been provided for destroying sessions and cookies resp. credentials for automatic login.
On iPad there is a problem with CKEDITOR, research shows that this is a known problem which will be fixed in the future release 5 of CKEDITOR, so far the problem is captured and a normal textarea is provided on iPad. The indicator of the problem is: var myeditor = CKEDITOR.replace(...); returns null. For users on iPad that means no formatting of text and copy from word-processing applications only transfers text and no formatting.


November 12, 2016
After doing a lot of experimenting and checking my decision is:
  • a vertical scrollbar on the page does the work automatically
  • additionally after resizing sometimes a second scrollbar on the content shows up, that seems to be by design of jQuery Mobile, I accept it
  • also from time to time after resizing a white space is displayed under the content area when scrolling, that too, seems to be by design of jqm and I accept it
  • no data is hidden when scrolling - that was my major concern
  • the resize with 100ms waiting is applied, but it does not that much
The decisive point was a misunderstanding of the page architecture of jQuery Mobile, the figures that I traced show:
  • window height - is the pysical height of the window that the app has
  • document height - is a virtual height including data that is not visible and needs scrolling
  • page height - is a virtual height, it is the same as document height
  • page outerHeight - is virtual and corresponds to page height + header height + footer heigth
  • content height - is a virtual height including the data not visible in scroll areas
  • content outerHeight - is virtual and has content padding in addition to content height
  • page padding top - corresponds to header height
  • page padding bottom - corresponds to footer height
Next point will be to change the cursor when the app is waiting. 

$("body").css("cursor", "progress");
and then back to normal again
$("body").css("cursor", "default");
November 11, 2016
the crollbar and the listview are positioned to the last element of the listview for pushmessages. That is done with a special function, easy and robust, third party scroll tools were too complicated and jQuery Mobile itself has weaknesses.
BUT: the scrollbar is now moving well, but the resize makes problems. I found
"Code in a resize handler should never rely on the number of times the handler is called. Depending on implementation, resize events can be sent continuously as the resizing is in progress (the typical behavior in Internet Explorer and WebKit-based browsers such as Safari and Chrome), or only once at the end of the resize operation (the typical behavior in some other browsers such as Opera)."
And this seems to be the root of the problem: continuous calls to the resize function with custom calculation makes it difficult to adapt.
If there is no custom logic for resize, then a second scrollbar appears and there is an overlay of footer and content area which hides the bottom part of the listview

The scrolling can be calculated but as collateral damage a second scrollbar appears. Therefore another try to iScroll together with iScrollView according to http://jsfiddle.net/Gajotres/CKSYJ/
- jquery.mobile.iscrollview.css
- jquery.mobile.iscrollview-pull.css
- iscroll.js

- jquery.mobile.iscrollview.js

November 10, 2016
http://stackoverflow.com/questions/11173589/best-way-to-create-nested-html-elements-with-jquery - one of the rare places where a special technique of declaring HTML in Javascript is discussed, append on the same level is explained. But append was not the best approach, insertBefore war much better.


        $("<span/>", {
            class: "ui-btn-icon-notext ui-icon-bullets",
            css: {
                "display": "inline-block",
                "position": "relative",
                "vertical-align": "middle"
            }
        }).insertBefore($("<span/>", {
            html: btext3
        }).appendTo($("<div/>", {}).appendTo(
            $("<td/>", {
                width: "20%",
                align: "center",
                css: {
                    "vertical-align": "middle",
                    "horizontal-align": "center"
                },
                click: function () {
                    $("body").pagecontainer("change", "#msgpage", {
                        transition: "flip"
                    });
                }
            }).appendTo(footerRowHash)

        )));


November 8, 2016
"Benutzererfahrung und Telemetrie im verbundenen Modus" or "diagnostic tracking service" might be a performance consumer, I stopped the service and deactivated it - it seems to be better now.
Regarding jQuery Mobile with elaborate layout of the footer I startet using table as layout-basis. Seems to be a good opportunity.
The JS Foundation got remarkable publicity, impressive members (IBM, Samsung) and jQuery Mobile is listed
And I check http://blog.axxg.de/js-top-10-jquery-mobile-snippets/
Position and size of buttons and icons in buttons in table elements is not convenient, so icons and text are directly shown in table elements http://stackoverflow.com/questions/9348729/jquery-mobile-navigation-bar-icon-size and http://jsfiddle.net/ezanker/Cj5rg/1/

  • button - not nice enough
  • icon plus text - ugly
  • span for icon and span for text - I give it a try
  • two rows, one for icons and one for text - if nothing else helps


November 7, 2016
The push-service is tested locally and now has to be implemented remote. Therefore the old book-application is removed and replaced by the push-service. Later the book-application will come back, written with the new framework.
The homepage will be adapted to that.

November 2, 2016
There were several problems in the layout of the footer. Finally the following solution was found:

  • for the browser
    • the footer has a text-line and a navbar with five buttons
    • the layout has a scrollbar beside header, content and footer, it the content needs it
  • for an app with Intel XDK
    • the footer cannot have a text-line, because it is presented beside the navbar and not above the navbar
    • the five buttons in the navbar must be calculated in width with px and assigned to .ui-block-a to .ui-block-e, then the buttons have equal size and cover the whole width of the footer
    • the scrollbar is presented aside of the content, it does not show beside header and footer

        if (appMode && appMode === true) {

            $("#" + actPageId + "navbar").navbar();

            var wnb = $(actFooter).width();

            var wne = (wnb / 5).toFixed() - 3;

            $("#" + actPageId + "navbar  .ui-block-a").width(wne + "px");

            $("#" + actPageId + "navbar  .ui-block-b").width(wne + "px");

            $("#" + actPageId + "navbar  .ui-block-c").width(wne + "px");

            $("#" + actPageId + "navbar  .ui-block-d").width(wne + "px");

            $("#" + actPageId + "navbar  .ui-block-e").width(wne + "px");

        } else {

            $("#" + actPageId + "navbar").navbar();

        }



The code has the following points:

  • appMode is set in the application, if true it's app with Intel XDK,  if false it's node.js and browser
  • actPageId is the string with the id of the activePage
  • "navbar" is the suffix for the id of the navbar


The click-behaviour was strange: when you click in an empty area of the content, where no click-event is defined, then the header or the footer of both hide and show. This behavior makes sense when you want to show pictures fullscreen. This behaviour is prevented by global configuration:


$.mobile.toolbar.prototype.options.updatePagePadding = false;

  $.mobile.toolbar.prototype.options.hideDuringFocus = "";

  $.mobile.toolbar.prototype.options.tapToggle = false;

October 24, 2016
Principles: use natural or logical keys and not system or technical keys like uuid, guid etc. - that makes it easier to synchronize local and remote data and it allows the same control and navigation within the data.
  • addDiaryRecord - finished, proven in display of local data
  • getLastDiaryEntry - programmed
  • getOneSingleDiaryEntryByIsoday - programmed
  • getOneSingleDiaryEntry - programmed
The calendar overview with the calendar control widget is necessary for the local mode, too. Solution: the existing page is used according to the rule: remote access first, if not possible, then local access. IndexedDB is read and the same structure of dates is passed as with ajax.

The initialization of the diary document is isolated in a function at least. 


An easier access to control of the testmode - so far it's in the sourcecode and it remains there at the moment.

October 23, 2016
Add data access to indexedDB in several functions according to priority principles:
  • addDiaryRecord - finished, proven in display of local data
  • getLastDiaryEntry
  • getOneSingleDiaryEntryByIsoday
  • getOneSingleDiaryEntry
Login with access to IndexedDB programmed and tested, it was necessary to program that first, because it's a prerequisite for add and get functions.

October, 22, 2016
Error on prev, next, swipeleft, swiperight - not saving the data
The prev-Button does not update the document. The same for next, swipeleft and swiperight. The Back-Button does update correctly.
Reason:
the backbutton is executed in the central controller of the pagecontainer on prevPage within an async construct that first envokes daysave and later does the navigation.
The other buttons are executed directly with daysave and the navigation in the callback function.
As all that did function before the local storage was introduced into the app - there is the point to start.
The optimization to navigate after save lead to a missing callback in the save function, the back-button was not disturbed because the async.waterfall compensated the error. The callback was added, everything works fine.
Automatic recognition of registration and former login in app-mode
The login with the browser uses a cookie to provide automatic login.
In an app the cookies are not available or only available with restrictions that make them useless for automatic login.
So when appMode === true the former login has to be controlled by using the indexedDB which is used in any case. The flow of control in the appMode is as follows:
  • index.html - references to js and css files and basic statements for initialization
  • app.js - initial module that is envoked by index.html
  • uisplash.js - module for checking central points 
    • check active server-connections (local or remote server)
    • check localization
    • check indexedDB availability
  • nta1010login.js - is envoked after uisplash was executed and the result was displayed for 7 seconds
The login-module is activated with the beforechange-function and checks the appMode. If it's the browser-Mode then the cookie is used for automatic login which directly goes to the Calendar-Display it the login data is available. In the browser-mode the login-data are checked against the server before the navigation goes to the Calendar-Display. If the login-data is not correct the user has to provide the login-data

In the appMode the beforechange function hat to check against the indexedDB with table or document type "USAGE". The automatic login is handled differently:
  • if no server is available, the it makes no sense to check the login against the server
  • it is checked, if old login-data is in indexedDB
    • if yes, then the login is passed and navigation goes to Calendar-Display
    • if not, then execution is stopped, because usage is only allowed it login has at least been done once after installation
  • if a server is available, then the login-check is done with the data from indexedDB and the processing cooresponds to the cookie-based automatic login
    • the login data is passed to the server
    • if the login is correct the navigation goes to the Calendar-Display
    • otherwise a login is required - the user may have been blocked in the meantime
















Wednesday, October 12, 2016

IndexedDB and MongoDB - local and remote updates

December 12, 2016 - reworked to the actual system state

The browser-processing uses ajax, node.js and MongoDB for data access and data storage.

The app uses:
  • local data access and data storage based on IndexedDB
  • additional remote data access and data storage based on ajax, node.js and MongoDB
Synchronization is done as follows:
  • in the browser there is no synchronization necessary
  • in the app you have
    • calendar: comparison of the dates with data in the local IndexedDB and the remote MongoDB when the calendar is shown - indicated by the color of the days in the calendar
    • day entry: when the user chooses a day to enter more data or just see the data, the system checks if there is a local record and a remote record, if both are there, a synchronization for the user and day is done
      • this synchronization at moment is only put to the server, if the user changes the record oder enters more data
    • mass-synchronization: the user has to choose this menue point, the system checks
      • for the last 100 days for a user
      • if a record is local and not remote, then it is transferred from local to remote
      • if a record is remote and not local, then it is transferred from remote to local
      • a comparison and synchronization in case a day has local and remote data is not done, that is only done when day-data are entered (see above)
If a user has a clear strategy to update his data only with the app, then nothing bad can happen, the system synchronizes when new data is entered and in case the remote system is not available the mass-synchronization can be activated - or the days are invoked one by on depending on the color in the calendar

The diagnosis of problems, when requests to read to change data go to the server will be handled in an expanded way:
  • try-catch for connectivity breakdown (get's added)
  • ajax-fail for problems within the connectivity, like false request path or false response data
  • ajax-done with no data as "special" result


The solution does not only require new code, but also new principles for the appMode:
  • getting data
    • if a server connection is available, read from server connection (AJAX, MongoDB) first
    • if appMode, read locally from IndexedDB 
    • check if the corresponding local data in IndexedDB has to be synchronized
      • always use logical keys and not technical keys (username and date and not UID's of any kind)
      • use hashcodes to determine, if synchronization is necessary
      • do the synchronization
      • if synchronization has been done and a server connection is available, write synchronized data to sever (AJAX, MongoDB)
      • if synchronization has beed done, write synchronized data to local indexedDB
    • put synchronized data to the ui
    • this way a smooth synchronization is done automatically and incrementally, transparent for the user
  • putting data
    • at first always put data locally (insert, update with merge) to IndexedDB
    • then put data remote, if a server connection is available (AJAX, MongoDB)
      • the remote server does synchronize and merge for updates
  • synchronization is done automatically in the background
    • the app sets a flag if the server connection is not available
    • a loop checks it the server connection is available again 
      • "keep alive" check
      • long waiting time, progressive from 1 minute to 20 minutes after 2 hours
    • if the server is available again then the synchronization is started automatically
    • the user just gets a message and a flag in the footer region to indicate the synchronization
Not all functions and data have to be supported locally. The critical functions for local and remote support are:
  • initDB - initializing functions for the database, the document store and indexed
  • login - a function that checks the userdata, locally that will be done against a cookie, but the login protocol data is stored in the database
  • putTagesdaten - a function to save the diary-data that has been entered
    • depending on the data it's insert or update respectively "upsert"
    • regarding the data content on the server a merge logic with aggregation of income, expenses and workhours is necessary, the app is single user in local mode
  • getCalendar - a function retrieving the dates, for which diary data is in the database, the dates are compared to the dates in IndexedDB - this can be used to synchronize
  • getTageListe - a function which retrieves detail data with certain selection logic
    • getTageListe - for a special date and username
    • getTageListe - for the day with data before a selection-date
    • getTageListe - for the day with data after a selection-date
    • getTageListe - for all data of the username 
    • the function for coreading will only be available on the browser
  • putUserData - will be a new function for users to change their password and their eMail address
  • getUserData - will be a new function for users to get their eMail address (the password is not transmitted from the server, but it's checked in the server)
Technically the code from node.js will be used as basis for the corresponding functions in the app.

The integration follows clear principles:
  • the calls to the existing functions on the server are already there as ajax invocations
  • these calls will be wrapped by async.waterfall, if that is not already the case
  • the async.waterfall gets new functions for the local data access and data storage
  • the functions with the ajax-calls get wrapped with conditions regarding server availability
The it's not a very big deal. The synchronization has already been programmed as manually started batch processing - this will be changed by automatic invocation after connection is reestablished after it was missing for a while. Classically it's a watchdog-mechanism.

Is smooth synchronization enough?
Smooth synchronization means that data is synchronized on demand of requests to the server. Two aspects:
  • when the calendar control is shown it can be supplied by the server or by IndexedDB - not with synchronization 
  • when a record is read it is read from server and from indexedDB - not yet with synchronization
What will be the best strategy?
  • calendar display can mark which records are not in indexedDB, they can be colored differently
  • automatic synchronization on calendar display can take some time, that will disturb the user
  • the user can click on dates, that require synchronization and that does the synchronization, even it no input was provided (meaning the dirty flag is set automatically)
    • but that makes synchronization a probabilistic matter
  • if a record is worked on, then it is synchronized in any case
A new synchronization logic is implemented, based on username and isoday.
  • Calendar-based synchronization
  • the getCalendar-API to the server gives extended information with a hashcode
  • the hashcode is calculated for indexedDB
  • comparision
  • action
    • the calendar control has new coloring
      • blue - data only on server
      • red - data only local
      • green - data is on server and local
    • priority is given to save data from local to the server
      • if red then the update to the server is called immediately, that's done in the thinktime of the user, at least it should be
  • Diary-entry based synchronization
    • hashcode-check is done when server is available by comparing, if different hashcodes, then an update to the server is done, the server merges and the reading of the server is repeated.
    • when the user updates, both systems get the latest version
Decision:

  • the getCalendar mechanism which compares the dates with data in IndexedDB and in the server will show a button for synchronization and the user can choose to do it
  • the synchronization will be done with the existing system - checking picture-synchronization will be done
ntasyncpage wird genutzt