Thursday, August 25, 2016

Listviews with scrollbars in jQuery Mobile

Abstract: on a smartphone scrollbars are not an issue, header and footer are fixed, the content is scrolled on the right. But for browser applications like upload files by developers, scrollbars are needed for different listviews. Various tools where checked without success, the jQuery Mobile roadmap is not sufficient, so I went back to the roots of HTML - and it works in the browser very well and easy together with grid-layout and listview of jQuery Mobile
August, 23, 2016 Rolf Eckertz

There are several overviews in the internet like  http://ourcodeworld.com/articles/read/179/top-7-best-custom-browser-scrollbar-javascript-and-jquery-plugins

I tested iscroolview, tinyscrollbar, nicescroll, iscrolljs, the documentation was not always the best for beginners and I like robust and easy solutions.

At this time iscrolljs is the favorite - there are a pdf and a lot of examples. iscrolljs is on github and has a special website.

The harmonization with jQuery Mobile was not that easy, no examples and no detailled instructions. So I had to find out with a step by step approach of putting things together:
  • the documentation says it's easy, just do:
    • var myScroll = new IScroll('#wrapper');
  • but that is not sufficient, you need style sheet directives too
  • I derived these from one of the examples
  • then I made the styplesheets more abstract, not dedicated to id's but to the classes wrapper and scroller
    • wrapper is the class of the outer div 
    • scroller is the class of the inner div in the wrapper
    • ul then starts the listview within the scroller
That gives code like:

<div id="X" class="wrapper">
<div id="Y" class="scroller">
<ul id=liste data-role="listview" data-inset="true">
<li>Pretty row 1</li>
</ul>
</div>
</div>

The next problems came when li-elements were added dynamically. Step by step:
  • when new IScroll('#X'); is executed, the li-Elements that are already assigned are shown
  • if you add more elements, they are not shown
  • in the documentation a setTimeout-Wrapper around new IScroll is recommended, but that is not enough, because listview("refresh") also has to be done. So far my solution is after dynamic adding li-elements:
setTimeout(function () {
myScroll.destroy();
myScroll = new IScroll('#X', { 
scrollbars: true,
mouseWheel: true,
interactiveScrollbars: true,
shrinkScrollbars: 'scale',
fadeScrollbars: true
});
$("#liste").listview("refresh");
    }, 10);

I did not manage a refresh on IScroll. The solution worked in a clean test-environment, but not in the complete solution environment. I found out: resizing the browser window somehow activated the scrollbars and the scrollwheel? The solution following the documentation: increased the waiting time in setTimeout to 100 and everything worked fine. With 100 the $("#liste").listview("refresh"); statement can be put before the setTimeout-statement.

BUT: this approach only works for one listview on a page so far, so I have to restart to build a multi-listview solution bottom up. After some experiments the result seems to me: it's not possible to have multiple listviews with their own scrollbars on one page, jQuery Mobile always tries to supply one scrollbar on the page-content level. The roadmap for jQuery Mobile points to release 1.7 for "Make panels scroll independently from page" and as we see 1.4.5 at the moment and little progress it will surely take a while for that feature.

On the other hand, the roadmap shows that there is a real problem with multiple scrollbars in jQuery Mobile.

Back to the roots in HTML scrollbars are easy:
  • you need a div as wrapper
  • the wrapper must have a fixed height
  • the scrollbar must be allowed
<div id="" style="overflow:scroll; height:400px;">


Sometimes decisions have to be made, here: back to the roots of HTML:
  • for smartphone and touchscreen jQuery Mobile is ok
  • for desktop there are restrictions and roadmap-points that are too far in the future
  • so: for desktop other solutions are necessary - or a special restriction to a subset of jQuery Mobile.

The screenshot shows: two listviews, two scrollbars - and just html:

<body style="overflow:hidden;">
...
<div class="list ui-block-a" style="overflow:auto; max-height:100vh;">
...
<div class="list ui-block-b" style="background-color:yellow; overflow:auto; max-height:100vh;">

So sometimes back to the roots is not the worst choice. On the long run the ui-code for my apps will be generated from declarative scripts, that way it's rather easy to do responsive design on the fly.

The solution finally has the following points:

  • pagecontainer - the beforechange-event is no longer used, instead transition is used
  • with transition there are no problems to calculate a fixed height for the block-elements that get the scrollbars
  • the 100vh-parameter did not work fine, because a lot of data is in an invisible area, with the calculated height between the form and the footer everything works fine
  • transition requires that the target page is already defined, otherwise it is not fired, but dynamic loading of the target page is possible
November 21, 2016
The understanding of screen, window, page, header, content and footer is deeper now, scrolling programatically to a special element can be done rather easily.



Tuesday, August 23, 2016

Select multiple files and upload

Abstract: there are different libraries for browser-uploads from local directories to node.js. Finally multer  has been used. Listviews with scrollbars are displayed to show the update-directories in node.js and the uploaded files.
August 24, 2016, Rolf Eckertz

The following functions are needed:
  • show update-directories that are already existing on the server
    • get directories with fs.readdirSync(dir).sort()
    • the update-directory must not be "read-only", is checked by fs.access - the api-documentation has errors, betterfs.accessSync(dirinfo.fullname, fs.W_OK )
      • BUT: since windows 7 directories are always read-only
      • so the update-directories are checked regarding the progress of work:
        • is there an upd-package.json-file
        • is there an upd-release.json-file
        • is there an upd-transfer.json-file
      • if an upd-release.json-file exists, then the directory is regarded "read-only" - no change to a package is allowed after release
    • the date of creation is got by fs.statSync(dir + "/" + dirs[i]);
    • the highest update-number has to be extracted from the directory-names, it's the default for the working directory, regex-is used
      • var m = dirinfo.name.match(/update(\d*)(.*)/);
      • m[1] is the number, m[2] ist the comment
  • show files in selected directory
    • the selected directory is either the default directory (the "last"), a new directory or a selected directory from the list of update-directories
    • the files of that directory have to be shown recursively with the sub-directory structure where they are located
    • it could be performance-critical, so asynchronous access to the file-information should be done
    • flat recursion is a convenient technology to do that: one central result-array and a control-index for the iterative resolution of subdirectories
  • define update-directory
    • accept default directory
    • select existing update-directory (not "read-only") in order to add new files 
    • define new update-directory (button)
      • a comment is entered - this was planed to be done with a popup, but there is a compatibility problem between jQuer 2.2.3 andiQuery Mobile 1.4.5 which makes popup impossible. I tried various third party plugins for popup, no direct success - so downgraded jQuery to 1.12.4 and it works fine (tried 2.2.4, same problem, tried 3.1.0 - desaster in the UI)
      • an AJAX-Function calculated the new update-number, builds the fullname of the directory and allocates the directory 
      • the user selects and uploads files to the new update-directory
  • select file from the local file-system
    • filter on file-type, not as easy as it should be
  • upload the file to the update-directory that has been selected or defined before
    • check and create sub-directories, if necessary
  • reminder: select multiple files in the browser, here: js-files - is possible, but not very convenient
  • show progress bar for the uploads (not highest priority)
  • give message for finishing the uploads and show listview with the files added
  • "finish and release" update-directory
    • in the server the files are put to the MongoDB, summarizing data about size and number of files are calculated and stored
    • the update-directory is marked "read only" on the server (for Windows)
https://developers.openshift.com/managing-your-applications/filesystem.html - this post shows where files are stored in openshift. The location directory comes with the property OPENSHIFT_DATA_DIR

There are several node.js Tools that support multiple file upload, formidable (https://coligo.io/building-ajax-file-uploader-with-node/) and multer (https://codeforgeek.com/2014/11/file-uploads-using-node-js/) are two of these. A little bit simpler seems https://www.npmjs.com/package/lite-uploader - but the server part is missing in the samples. 
Multer seems to be the favorite. https://www.codementor.io/tips/9172397814/setup-file-uploading-in-an-express-js-application-using-multer-js gives a good tutorial that will be used as starter.
BUT the app and the browser-application are written als SPA-single page app and that means that file data has to be treated by AJAX. http://stackoverflow.com/questions/36092622/file-upload-to-nodejs-using-ajax gives some idea for that approach.
It seems to be standard to have form and submit button and transform it with jQueryForm Plugin from https://github.com/malsup/form and http://malsup.com/jquery/form/#file-upload with additional examples.
https://codeforgeek.com/2014/11/ajax-file-upload-node-js/ finally is the basic approach to the solution based on jQueryForm and Multer https://github.com/expressjs/multer - one file after the other.

For directory- and file-attributes https://www.npmjs.com/package/winattr is used, seems to be the most convenient solution.
The next technical aspect is dynamic loading of js code






secure login and session management

November 16, 2016
Additional details for destroying sessions and cookies resp. credentials for automatic login within a new logout function

The following functions are necessary for a secure login and session management:
                       if (req.headers['x-forwarded-proto'] == 'http') {
                           res.redirect('https://' + req.headers.host + req.path);
                     }

    • see how to integrate this with app.use
    • and this is my solution, https for productitve use if there is a proxy, otherwise local test with http, at the time I test:

app.use(function (req, res, next) {

  if (req.headers['x-forwarded-proto'] == 'http') {
    console.log("REDIRECT TO HTTPS");

    res.redirect('https://' + req.headers.host + req.path);

  } else {

    console.log("NO REDIRECT TO HTTPS");

    next();

  }

});
    • res.redirect doesn't work with POST - so the redirect was modified to use the host and no longer the path, that forced login with https as desired
    • I want to mention, that the above statements must be placed before:
      • app.use(express.static(path.join(__dirname, 'static')));
    • the app is another story - it must use https always, as there are only AJAX-requests
  • cookies must be allowed for session management
    • the session management is based on session cookies
    • if cookies are not allowed the login is not possible
  • login-page as start-page
    • the system must force the loginpage, as long as there is no valid login, this is done by configuration options for jQuery Mobile 1.4.5 (go to the end of the discussion!)
    • additionally every request checks, if a session is active, if not, login is enforced
    • basics for Login and Session-Management: 
    • the system delivers a captcha additionally to the entry fields
      • https://www.npmjs.com/package/captchapng is used
      • later md5 will be additionally used to encrypt the captcha value on the server in the session and to encrypt the user input on the server too before it is compared
    • the user enters
      • user-name, usually the email-address
      • password
      • captcha (later)
    • the data are checked
      • check against the user-table
      • check the password, it's simply hashed with MD5 and will be enhanced to unique hash later
      • check the user-type, admins have more rights in user-management than others
    • When the USER-table is empty the admin is automatically directed to the user-entry, after that the system functions like expected, every user must be installed by an admin to be able to work - normal users can enter new normal users only.
  • automatic login with cookie
    • if the user uses "remember me" then a cookie stored the credentials of the user
    • the credentials are stored encrypted in the cookie
    • on invocation the system checks the cookie and does the login automatically
    • if the user no longer wishes the automatic login, then he goes back to the login screen and sets off "remember me" and does one more login, that deletes the cookie with the credentials - or he does a logout
  • automatic login with indexedDB
    • in an app the handling of cookies is rather different depending on the platform and not corresponding to the handling in the browser
    • therefore the credentials of an app are not stored in a cookie, they are stored in indexedDB accordingly (and encrypted)
  • logout
    • a logout-function has been provided
    • in the server req.session.destroy(function (err) { ... }); is used to finish the session
    • but that is not enough - the server additionally has to destroy the client side cookie: res.clearCookie("server-session-cookie-id");
    • as on the client the credentials are stored in an additional cookie, this cookie is destroyed in the client by: 
      • Cookies.remove("username", {
      •       domain: location.hostname
      • });
    • the destruction of the credentials in indexedDB is done clientside too.

Automatic Software Updates for Hybrid Apps

Abstract: Doing Updates by new installation of an app is not safe and convenient for users. An automatic update system has to be supplied. A developer makes an update package und puts it on the server, the app checks: "new update" and does the update, the user has nothing to do. Technical challenges: Load files from browser to node.js, nice interface for the developer, download to device, install in the device.
August 24, 2016, Rolf Eckertz

Ninas Travel App is programmed in JavaScript - and the idea of dynamically adding or exchanging javascript code goes back to the begin of the internet and browser programming.

Meanwhile jQuery made it very easy to get JavaScript-code into an app. You can choose getting it from an ajax call or from a string. And you can supply the string however you want, from a file, a database oder code-generation - all these techniques have been developed long time ago.

Ninas Travel App is written using isomorphic code, that means: code can be used in the app, in the browser and in the node.js backend. This is done by a special kind of modularization that is state of the art in javascript-programming and impemented in a lot of libraries already.

I checked of lot of standard solutions like commonJS, requireJS and more. My opinion is that these solutions do force me into restrictions that I do not want. On the other hand a solution just based on jQuery and usage of IndexedDB in the App and MongoDB in the server-backend can be build easily and in a robust way - that's what I will do.

The backend has an update system based on openshift from red hat cloud, GIT does the source control an the deployment on the server an the restart of the server. This will also be used in the future. The browser-functions are delivered from the node.js backend, so the browser will not get dynamic updates as it will be up to date after every restart.

The core of the automatic software updates for hybrid apps is on node.js. There is a central updates-directory in which all the updates exist als sub-directories with the name-schema:

  • update<Number><-optional comment>
The status of an update-subdirectory can directly be seen by the following files:
  • if an upd-package.json file exists, then the developer has finished his work
  • if an upd-release.json file exists, the the quality assurance has finshed work with release of the package, that means: no further changes may be done to the directory
  • if an upd-transfer.json file exists, then apps on devices have requested and downloaded the update. This is documented in the file

So there will be the following components and functions:
  • app modules are developed under Intel XDK
  • when these modules are testet as app they are also tested as browser-modules
  • the server gets the following functions as browser and backend-functions
    • selection of modules (js-Files) for dynamic update 
    • entry of documentation for the users
    • set a release-timestamp for the uploaded modules
    • storage of the selected modules into MongoDB with the release
    • create a release.json-file documenting the release and a package.json-file documenting the contents in the update-directory
  • the apps get the following new functions
    • check if the server has new updates and download these updates into the local indexedDB (done by standard ajax-calls with access to MongoDB)
    • check if the transfer of the updates was completely done and consistent
    • set markers for the next start of the app
    • give Information to the user to allow a restart of the app
    • dynamically load the new js-files
    • send feedback to the server that the update has been implemented for use
    • in the server the upload of the updates is documented in the transfer.json-file in the update-directory and in MongoDB
  • the server gets the following functions as Web-Services (AJAX)
    • process and response to checkForUpdate Request
    • process and response to getForUpdate Request (sends the code)
    • process and response to feedbackForUpdate Request
Future releases are planed, that will check the consistency of the subdirectory and the package in it by hash-codes and data-contents of the json-files.