speech recognition

Speech recognition and speech to text (STT) are convenience at it's best for the user of small devices.

I did research on speech recognition for my book project, in case the barcode-reader could not recognize the ISBN barcode on a book it was possible to dictate the numbers of the ISBN - one by one.

Today the circumstances are a little bit better.

photos - select, display, store thumbnail

December 11, 2016 - optimized code with closure and additional file information

There are some steps necessary to present and store pictures in the diary with browser or app:

• select pictures from file system - FileReader HTML5 is necessary to select pictures
• create thumbnail, crop the thumbnail image and create base64 string for this compressed or reduced picture - crop is necessary because thumbnail is an optical effect and not a compression of the data of the picture - canvas HTML5 is necessary for this function
• show the thumbnail to the user - plain HTML with base64 string as image src
• store the thumbnail in the database - plain base64 string
• retrieve the thumbnail from the database and show it again - base64 string is image src again

The best way to select pictures from the file system I found in http://stackoverflow.com/questions/27254735/filereader-onload-with-result-and-parameter and the fiddle http://jsfiddle.net/sahilbatla/hjk3u2ee/ - the code uses closures. You get a clean loop on the files and processing the files is easy. But jshint still marks the function in the closure, because it's in a loop, although it's save.

FileReader is not available on Safari under Windows 10.
Photos are an important part for a diary. In
http://codepedia.info/editor-example/preview-image-before-upload-jquery-example/ there is an introduction with sample code for using HTML5 for file upload.

The images are displayed as thumbnail.

The corresponding html-source shows that the image is referenced with 

<img src="data:image/jpeg;base64,<base64-String>>

And this is a very good opportunitiy to upload the picture to the server or the local database.

https://github.com/nodeca/pica seems to be a good approach but the solution is large regarding the code that has to be downloaded to the client. Under node.js that would not be a problem.

Based on canvas creating a thumbnail is easy and efficient:

http://stackoverflow.com/questions/15990946/resize-image-before-upload-convert-canvas-to-a-file-object gives some background
http://jsfiddle.net/cL3hapL4/2/ shows how to do it, you can easily add some try catch block structure - just in case there is an old browser
http://stackoverflow.com/questions/15685698/getting-binary-base64-data-from-html5-canvas-readasbinarystring shows canvas.toDataURL("image/jpeg"); which will save the reduced photo.

The base64 String still has some size, so compression can be the next step.

http://pieroxy.net/blog/pages/lz-string/demo.html is a very good solution for compression, but then you have a blob - which can cause a lot of problems when store to a client database, see https://github.com/nolanlawson/state-of-binary-data-in-the-browser

My solution:

• HTML5 FileReader is used to select pictures from the local storage
• html img is used to show the picture, src comes from the FileReader as base64-String
• canvas is used to create and save the thumbnail
• getContext("2d");
• drawImage
• toDataURL("image/jpeg", 0.50); - uses the quality downsizing

The resulting base64-String from toDataURL is impressingly small.

http://stackoverflow.com/questions/28538913/crop-an-image-displayed-in-a-canvas shows a simple solution to crop with canvas. For the resizing with canvas the canvas has to be set to the target width and height, before the resize is done, then everythin works fine - code to come
http://stackoverflow.com/questions/27254735/filereader-onload-with-result-and-parameter shows how the inner function in the loop over the the imagefiles can be formulated as closures, then jshint will no longer put a mark on it.

create and export pdf

Creating and exporting pdf as file or print output is a necessary functionality.

• pdfkit is a good library for generating pdf output and it will be testet first
• pdfmake is a wrapper around pdfkit which make it simpler to define pdf reports and it also provides the opportunity to use it in the client

Both solutions will be testet. The documentation so far lacks the printing of footers and of page-numbers, but I'm sure, there will be solutions.

The first usecase is creating a pdf file for the diary data of one day. To support that, a print icon will be supplied for the diary entry.

A first test with pdfmake from the browser with download of the generated pdf-file was very promising.

Some background is given in http://gonehybrid.com/how-to-create-and-display-a-pdf-file-in-your-ionic-app/

PDF.js is a pdf viewer for usage in the browser and in an app.

The printing of the pdf documents seems to be a little bit more complicated. There are various approaches working with cordova:

• the approach sarahgoldman/cordova-print-pdf-plugin prints pdf after conversion to base64 and  communicates with AirPrint for iOS and Android Printing APIs
• the approach katzer/cordova-plugin-printer can print html-content from iOS, Android and Windows Universal apps

Generating reports with pdfmake has the following steps:

• create basic docDefinition als JSON
• add more data to the docDefinition with JavaScript-Statements
• create the pdf-File

You have to get accustomed to using the styles, but once you get it, it's very easy.

A challenge is using richt text from CKEditor, there are some html-tags in the text, so a conversion is necessary. 

• <br> to /n
• <b> and </b> to a new paragraph with style strong
• list-header and list-elements have to be converted accordingly

On github there is an approach using the DOM for the conversion - seems to be promising. In https://github.com/bpampuch/pdfmake/issues/205 there is a discussion to this challenge and http://jsfiddle.net/mychn9bo/75/  shows a good approach.

As the diary app has various data like text, richt text and tables in the MongoDB and IndexedDB documents, it seems to be good to use a modular approach that generates the paragraphs for the pdf from the data-elements.

captcha - captchapng - pnglib and problemsolutions

Using a captcha to make sure real humans do a registry or a login is not a bad idea.
There are three approaches to my knowledge

• using a number or some letters and showing a "strange" picture representing it, the user has to enter the number or the letters
• using a formula, the user has to do the calculation and enter the result
• using a fragmented picture and the user has to choose the fragments that fit to a question (a google service can be used for that)

I preferred the first approach with the strange picture. I found captchapng, a solution that only uses javascript and not installation of other applications aside of node.js and no big C++ addon ...

BUT: captchapng has a statement that is recognized as error by the V8 javascript engine that is running on openshift as my node.js server. 

The problem: 

• there are branches to captchapng and pnglib with fixes to the problem
• but theses branches are not releases to new packages
• a fix in the local source of pnglib.js is ignored by node.js on openshift, the implementation seems to strictly reagard package.json directives and not the sources provided via git.
• experiments with changes to package.json on the app-level and the module-level were not successful
• I simplified:
• captchapng.js is in the root, where I have server.js
• pnglib.js is in the root, where I have server.js
• server.js references require("./captchapng.js");
• catpchapng.js references require ("./pnglib.js");
• additionally var myself is defined for strict mode
• and the correction in pgnlib.js is done:
• return "\x89PNG\r\n\x1a\n" + this.buffer.join('');
• and everything is fine

The error message referes to 

octal escape sequence are deprecated in es5, and cause error when pnglib is run or bundled in 'strict mode'

It would be great it the npm version of captchapng would be updates to directly support the corrected solution.

advanced functions

November 26,2016 - written
November 27, 2016 - priorities and status
December 9, 2016 - priorities and new points
December 16, 2016 - BERICHT will be changed to arry

The functions for the user and the ui are pretty much finished, the administrator functions are secured and a review of the blog written so far shows, there have been more ideas beyond basic functionality and security.

• Security
• open - automatic upload from server server to app (MongoDB to IndexedDB) when user logs in, so far there is only an automatic download from app to server with login and entering new data
• done - captcha for registration of new users, uses captchapng and pnglib
• return "\211PNG\r\n" + "\x1a" + "\n" + this.buffer.join(''); could be fixed
• later - captcha for login from app
• open - simple DOS-attack-recognition
• later - encryption of data in the server-database
• done - automatic login with cookie or indexedDB from client
• system availability
• done - automatic refresh of push-server connectivity, retry has to use healthcheck, not pull with longwait
• later - buffering messages when connection is busy
• open - save chat to database on the server (not available without connection)
• later - react to suspend/resume in appMode
• User convenience
• done - show aggregated data, income and expenses aggregated to months with drilldown to details
• done - show photos from the local storage, calculate thumbnail and store the sumbnail in the database, new section in the diary entry
• open - more data on photos to be stored (may be origin, timestamp)
• later - option to store the local photo to a cloud server
• open - change from BERICHT as string to BERICHT as array of structures with text and an inner array for comments to the original diary entry. 
• open - change or delete data, this requirement is hard to do, because at the moment concurrent access is allowed and a merge logic is done on the server to synchronize concurrent updates, that only can give additional text-data or additional rows in the tables for income, expenses and time-entries. Corrections to data in the tables have to be done by corrective records, e.g.:
• an income of 100 had been entered
• that was an error, 10 would have been correct
• so a new row has to be entered with -90 als corrective row, then sum will be correct.
• it's also possible to enter two corrective records, one with -100 to compensate the false rows and one row with 10 as correct value.
• open - filter on different channels
• open - user groups on channels
• open - allow valuation of messages (aggree, disagree,  don't understand, ignore)
• open - allow comment on messages
• open - mark diary entries for additional publishing as message (chat)
• later - show merged data of the own diary and the diaries with coreader-allowance
• later - allow comment on diary entry in the merged display of data
• open - pdf-export, a basic test has been implemented, the output of a pdf does not work on every device, more tests are needed, before the complete contents of the diary will be processed. The conversion of the html-tags that are provided by CKEditor to pdfmake compatible tags is under research and construction
• eMail export
• camera and picture stores
• translation
• speech to text
• development system and contents of my book 
• why a development system
• what is a program
• what is programming
• what is programmer productivity
• what is so special in app- and web-programming
• app-design
• pages
• functions
• data and JSON-objects
• data and ui-objects
• data-mappings
• callbacks
• async.waterfall
• async.parallel
• users, roles and rights to functions and api's
• header
• standard functions
• flyin menues
• content
• segment, grid-cell
• listpanels
• list
• collapsible
• accordeon
• responsive table
• entrypanels
• fieldsets
• fieldtypes
• declarative field definition (div-label-input) in imperative way
• mappings of ui, document and no-sql database
• footer
• standard functions
• dynamic functions
• general functions
• uihelper
• uisystem
• new applications
• homepage administration and authoring
• company
• news
• calendar
• product catalog
• references
• homepage presentation
• http-server
• backend-server
• chat-server

responsive design

Input-fields can easily be defined for responsive design:

        $("#reguserForm")

        .append('<div />', {

            class: "ui-field-contain"

        }).append($('<input />', {

            type: "password",

            id: "reguserpassword1",

            disabled: false

        }).append($('<label/>', {

            for: 'reguserpassword1',

            text: "Passwort (Kontrolleingabe)"

        })));

This field container approach allows a compact coding, on a small screen the label is positioned above the input field, on a wider screen the label is positioned aside of the input field.

Several input fields with their labels can be aggregated under one field container.

So far also tables have been used:

     $("<table />", {

            id: tblid,

            width: "100%",

            "data-role": "table",

            class: "ui-responsive",

            css: {

                "table-layout": "fixed"

            }

        }).appendTo("#" + contid);

The definition is easy, thead and tbody have to be used, otherwise they are added automatically. Then the directive class: "ui-responsive" is sufficient for the responsive layout:

• on wider screens the tables are shown "normally"
• on smaller screnns the tables are converted, every column of a row, that means every cell of the table gets an own row with a label above the input or output field of the cell.

The "table-layout": "fixed" is necessary for the definition on the cell level:

             css: {

                "word-wrap": "break-word"

            }

This makes sure that the layout is not changed by the content of the cells and that the content of a cell is shown completely.

The header and the footer are fixed with "data-position": "fixed" and that is enough to keep header and footer in their place, when the screen height resp. window-height are changed.

But a change in the content of header and footer can cause that the height of the header or the footer changes. This does not automatically lead to a recalculation and new positioning of the content. That' why custom code has been developed to do the resizing of the content. That can not be done easily, instead the page parameters have to be changed, because the header-height is the page padding-top and the footer-height is the padding-bottom of the page.

That was the view back, now to the next challenges:

• example registration page
• the content is a sequence of input fields and checkboxes and a button
• vertical alignment is sufficient
• on a wider screen the input-fields are too wide - that looks ugly
• using max-width on the content-level and center on the page level gives a nicer layout

The solution for the content is quite easy - the yellow color is just for testing and demonstration:

css: {
    "max-width": "750px",
    "background-color": "yellow",
    "margin-left": "auto",
    "margin-right": "auto"

}

That looks not bad, when the content should be displayed with maximum width and the screen is very large, a grid layout with two columns can be used. In case of the diary:

• first grid cell: ratings and report
• second grid cell: income, expenses and work-time

this will be applied in the next step

simplify and beautify

The turnaround of data from the user to the local database and the remote database is robust and stable, the chat function is robust and stable and the ui is more stable and boring than exciting and attractive.

So the time has come to make the ui more exciting and more attractive.

General Points
The user-messages must be cleared after some time or when a page is navigated - there are some design-questions to clear and decide, before this can be done.

The splash screen
The splash-screen is shown for a maximum of 2 seconds and it has only technical data. Enhancements:

• colors unified
• shorter text - still work to be done, there should be a control-display of the systemdata, that can be displayed on purpose
• mouse indication system activity
• no redundant checking of system values and abilities

login page

The system information is displayed redundantly, better: 

• show only information relevant to the user (and not for the administrator)
• cookies must be allowed and are stored
• geo-location should be allowed, but that is no must
• active server
• active push server

register new user page

there are some enhancements

• the password has to have 6 to 20 digits, at least one small letter, one capital letter, one number
• http://www.w3resource.com/javascript/form/password-validation.php
• the password has to be entered twice
• copy and paste is not allowed for the second password entry
• the double input of the password is checked
• checkboxes are provided for the following acknowledgements
• the new user must accept the usage of cookies - otherwise the application cannot work correctly in session management (mandatory)
• the new user has to accept the storage of credentials for automatic login - otherwise the user has to reenter his credentials every time he wants to use the application (optional)
• the new user has to accept the testmode with restrictions to the content he is entering (mandatory)
• the imperative definition for this is a little bit tricky, but "easy if you know how"

$("#reguserForm")

.append($("<div/>", {

    class: "ui-field-contain"

})

.append($('<input />', {

    'type': 'checkbox',

    'id': 'checkCookies',

    'name': 'checkCookiesName'

}))

.append(

    $('<label />', {

    'for': 'checkCookies',

    text: "Cookies werden akzeptiert"

}))

.append($('<input />', {

    'type': 'checkbox',

    'id': 'checkAutoLogin',

    'name': 'checkAutoLoginName'

}))

.append(

    $('<label />', {

    'for': 'checkAutoLogin',

    text: "Automatischer Login"

}))

.append($('<input />', {

    'type': 'checkbox',

    'id': 'checkTestModus',

    'name': 'checkTestModusName'

}))

.append(

    $('<label />', {

    'for': 'checkTestModus',

    text: "Testmodus, vorsicht mit personenbezogenen Daten"

}))

);

 if ($("#checkTestModus").prop("checked")) { ...

• the username has to be checked
• when a new user tries to register, the eventually existing credentials of a former user have to be deleted, either in the cookies or in a local IndexedDB
• also the messages to the former user have to be deleted
• a newly registered user is directly passed to the start page, which is the calendar, he does not have to reenter his credentials
• a new username must be really new and not exist already in the database
• the cursor should be positioned to the first field and this field should be visible
• the page has a lot of data and input fields, so the button sometimes is hidden by the footer, specially when longer error messages are displayed in the footer for the user, that has been taken care of

// resize according to the size of the message

var actPage = $('body').pagecontainer("getActivePage");

var hh = $(actPage).find("[data-role=header]").outerHeight() || $(actPage).find("[data-role=header]").height();

var fh = $(actPage).find("[data-role=footer]").outerHeight() || $(actPage).find("[data-role=footer]").height();

$("#reguserpage").css("padding-bottom", fh);

var ch = $(actPage).outerHeight() - hh - fh;

// Cursor position to wrong field

$('#reguserpassword').focus();

• the code is for resizing according to the height of the footer and for cursor positioning - and it's pretty generic - but the global parameter for footer adaptation will be checked once more - and I have to admit that the problems came from parallel updates to the layout of different pages (the push message page was updated in the layout directly). Layout updates shall only be done on the active page.
• a conflict was there because messages from the registration where send to the push server, directly received and that lead to an interaction between the registration page and the push-/chat-page regarding positioning the scroll-position. Solution: rearranging the scrollposition in the push page is only done, when the push-page is the active page - that solved the problem
• when resizing, sometimes a second vertical scrollbar appears on the content, the first is on the page and that's ok. To avoid the second scrollbar I use on the content definition:
• css {"overflow": "hidden"}

calendar page

The calendar page can remain unchanged, there is a minor point regarding resizing. The datepicker widget which is used does not scale automatically, but a scaling effect can be used by changing the font-size, a smaller font-size makes the shape of the datepicker smaller accordingly.

This point is postponed.

The calendar page has a lot of additional functions that cannot be seen directly:

• getting all the dates with diary data from indexedDB
• getting all the dates with diary data from the server
• comparison of local and remote data
• check the local data regarding the username, if the user has changed, the old data is deleted for privacy reasons

Special points are:

• The color in the calendar fields for the dates is once more controlled, the colorization of the screen is removed, because it was used for testing reasons.
• blue - there are no local data but server data
• red - the data is local and not on the server, local data has to be saved in the remote server, this is done, when connectivity is there, it only applies to the app (Android, iOS, Windows 10)
• green - there are local data and data on the server for that day
• no color - no data available
• yellow - "today" and no data yet available
• the colors are assigned via css classes
• the globalization so far is prepared for Germany and Australia - but is is not configurable. The configuration will be done later together with the "first time configuration"

diary entry
the entry of diary data is rather sophisticated.
There are some details to enhance:

• display of existing entries - the layout has to be a little bit optimized, more compact - may come later
• entry of text is done with CKEditor or on iPad with normal textarea, the app checks it CKEditor is working and the fallback is the textarea
• income, expenses and worktime are collected in tables, the only point is that in 2017 income from work and travel (of course from the work part) is taxed in Australia with 19% - a design for that still has to be made
• rating of health, weather and mood is done with emoticons at the moment, an experiment with a slider was done, but is not implemented now

push messages
Some details have been enhanced:

• recognition of push-messages coming from the same username, these are not show to avoid cascading effects
• correct counting of incoming push-messages and updates to the footer 
• show push-messages in the message line of the footer, there the text has to be shortened and there will be a link to the push and chat messages from the footer, the following details where fixed

$("#pushmessage").val("");
$("#pushmessage").css({'height':'auto','overflow-y':'hidden'}).height($("#pushmessage").scrollHeight);

Surprisingly it was not easy to reset the message textarea height to one line. Another critical point is positioning the listview to the last element in the push and chat client. That has to be done when a new message is entered or a new message is received while the page is the active page.
The solution had to make a difference between app-mode and browser-mode.

In browser-mode: // $("body").scrollTop(totop);
$(window).scrollTop(totop);

is very easy, in app-mode the logic uses parentScroll and an iteration. But this does not work in appMode and there are Problems on the iPad - some work to do

Responsive Design
on the level of elementary input and of tables the responsive design has been implemented, now it's time to regard the general layout of applications - separate chapter.

from push-messages to chat

Push messages based on channels is programmed and tested - a good foundation for chats.

For chats you need:

• a register of users or groups of users with whom you want to chat
• that pretty much resembles the co-reader function that is already implemented
• and channels too can be used as group-definition
• a special chat dialog has to be programmed, that can be integrated to the display of push messages that already exists
• a comment-function to diary entries that open a push dialog - would also be a good idea for later enhancements

In an app with touchscreen and without physical keyboard the layout changes when the keyboard is displayed. That will be done automatically.
A study of existing applications with chat functions show:

• every application is a little bit different in the layout and the user navigation
• the old chat protocol is above the entry field for a new chat entry - that's standard
• in detail there are variations:
• the entry field is part of the footer and has a fixed position within the footer
• the entry field overwrites the footer 
• the entry field follows the text and is not part of the footer
• when the entry field is activated - for touch displays
• entry field and footer go to the top to give room to the virtual keyboard
• entry field goes to the top to give room to the virtual keyboard and footer is hidden

It's not possible to cover all the layouts of all the popular chat applications (like facebook, twitter, whatsapp, skype), so Ninas Travel App gets it's own layout with the intention of intuitive usability.

download, activation, configuration and updates with levels

the software has four levels:

• level 1: the system-core is always downloaded and it provides the functions for the activation and the configuration
• level 2: the application-core is always downloaded, it has the typical functions for the normal user
• level 3: the application-options can be chosen by the configuration depending on the user-license
• level 4: customized application-options can be chosen depending on the availability of customized applications, this level will be provided later

Level 1 and 2 will be delivered compressed (minified) and obfuscated, that guarantees a fast download and fast availability and it enforces security.

Level 3 is downloaded and installed on demand. The same will be with level 4 when it will be implemented.

push services - logging, messaging, updates

Push-Services are able to inform the active users about urgend messages. The core of a push-service is a message queue. Senders are putting new messages into the message queue and receivers are getting messages from the message queue. Of course the user-applications are not directly working with the message queue, they use have API's accordingly.

The senders into the message queue can be:

• log protocols of applications
• from a node.js server
• from a browser
• from an app
• messages entered by users
• from a browser
• from an app
• updates for application
• from administrators, packed in the browser
• send by the server

The receivers can be users with a browser or users of an app.

The receivers can define filters - they may not be interested in every kind of messages - the sender too can restrict the potential receivers by using the co-reader functionality or defining other filters. REMARK: the first edition of push services will not have filters, except on channels.

A good metaphor for the design of a push service is a "channel". Channels bundle different kinds of messages, a starter will be:

• userchannel - messages from users for other users
• systemchannel - messages from the administrator to all active users, e.g. to warn from a shutdown
• loggingchannel - messages from applications, intended for analysis by developers
• updatechannel - messages from the administrator containing update packages that will be installed silently and automatically, primarily for app-users but may be for browser-users too (although a message to refesh the browser would have the same effect)

A dev-op-question is, if the push-service should be installed as external cloud service, as integrated part of the application or as separate installation side by side to ninas travel app server. The starter will be the separat server with some api's - and that means cross domain access must be allowed for the browser and the app. 
The api's for the applications are:

• putmessage (former: putlog, POST) - client sends message to the push-server, the server puts the message into the queue and begins pushing
• pullmessage (former: pull, GET) - client requests new messages, a longwait is used to delay the response, if no messages are available, a longwait is done and if the client then receives no message he has to call pullmessage again, the client tool can use a longwait for polling.

A message that is transferred has the following data:

• channel - primary filter for messages, point-notation possible to differentiate
• level1 - title or short form of the message
• level2 - long form of the message
• severity - numeric, to be defined, serious starting at 3
• timestamp - timestamp from the client, numeric
• new Date().getTime()
• tscreated - timestamp from the server, numeric
• new Date().getTime()
• ip - address of the client, calculated on node.js
• username - from the client, sender

The coding on the server is rather straightforward, on the client it's more complicated, because it shall not interfere with the rest of the app or the browser application and the longpolling has to be executed on the client.

The basic solution of the past only knew one channel als filter, now a list of channels has to be processed.

The messages received have to be stored locally in the Browser or the App as the server only delivers new messages after the initial request. 

And:

• a counter ("#pushcount" for pushmessageCount) for the pushmessages has to be implemented and updated
• if browser-mode the message has to be displayed in the footer area
• otherwise in app-mode a temporary popup is necessary
• and the message goes into the list with prependTo as "last in first out", that was the first approach, but the usability is better if the newest messages show up in the bottom above the entry area for chats
• additionally the state of the push-server connection is displayed with color
• green - connected
• yellow - trying to reconnect
• red - connection is lost

The layout of display for chat will be:

• list of messages on top
• position to last element, when new element is added
• http://www.isummation.com/blog/scroll-to-ui-element-in-jquery-mobile-framework/
• $.mobile.silentScroll($(target + ' li:last').offset().top);
• the test with the entry-field below the list still has to be done, the list above the footer is working well
• entry area under the list area

This approach was not successful in the layout on different platforms. The entry area was put into the footer and then the scroll of the list to the last element did not work as expected.
http://stackoverflow.com/questions/6215779/scroll-if-element-is-not-visible
Regarding updates some design point already have been worked out:

• automatic software update for hybrid apps
• dynamic loading and executing js files

The basic Ideas for downloading the updates are: 

• using a special channel for downloading the updates
• the channel is activated when an app registers to the server
• when an update arrives
• the update is stored
• the update is checked if the activation is flagged
• when the activation is flagged the updates are done
• the details will be specified later

Chat will be another function based on push messages. Chat too is specified separately.


more to come