In the first article of this series I summarized which phases and steps we may have to cover during an Ajax driven file upload procedure for a ZIP container file. See:

CSV file upload with ZIP containers, jQuery, Ajax and PHP 5.4 progress tracking – I

In this second article, I shall describe in more detail what has to be done during the first step identified for Phase I – the file transfer phase, where a ZIP container file is to be sent over the Internet from the client to the server. We look at the client side and describe some special settings and “tricks” which are required to enable a working and parallel transfer of the ZIP container file and other parameter data to the server via jQuery’s Ajax interface.

Note that the information given below should be regarded as a collection of suggestions. The code snippets are enhanced to make them understandable. Everything can of course be programmed in a different and more efficient way.

We start with some minor preparations to take care of:

Naming conventions for the CSV-files of the Zip container and a related database table

The server must get a chance to distinguish between the required actions for the handling of each of the CSV files in the transferred Zip container. We do this by setting up a small table “fileToTbl” in the LAMP server’s database. We use this table to associate a file name with a target table of the database. A file “alpha.csv” may such be associated with a DB table named “alpha”.

If there are several files referring to a split sequence of data for one and the same target database table the CSV file names may get an integer suffix like “alpha_1.csv, alpha_2.csv, …” – which indicates a loading sequence. The server can extract relevant parts of the file names by the PHP function “explode()” during the treatment of the transferred files.

The table “fileToTbl” may furthermore contain a column “onr” which via an integer defines the standard order by which all possibly received files should be imported into the database. Thus, typical columns would be:

nr, file_name, db_table_name, onr

It is the user’s responsibility to fill only CSV files with the defined names into the ZIP-container to be uploaded. If during ZIP file extraction files are detected which do not fulfill the naming conventions the server should issue a warning in its Ajax response.

Note that although the table “fileToTbl” includes information about all possible files and their names, the number of CSV files send in a specific upload process may be much lower than the maximum possible number. The user may want to update only some input data tables.

Control array for a database import pipeline

The previously described measure enables the server to build up an ordered “pipeline” for loading the received CSV files sequentially into their target tables in Phases II, III, …. When the target PHP program of Phase I server extracts the files from the transferred ZIP-container it can analyze the file names and build up an ordered sequence for the import according to the information found in the table “fileToTbl”.

The information about the loading sequence can be encoded into an array. We call this array “DB Import Control array”, short DBIC-array. It can and will be updated at the end of each Ajax phase and will also exchanged with the client. The (associative) BDIC array should contain information about which files have already been loaded, which files still are to be loaded
and if an error occured. Therefore, elements of the array could be:

dbic[i][‘fname’] = ‘Name_of_the_i-th_file’;
dbic[i][‘fsize’] = ‘Size_of_the_i-th_file’;
dbic[i][‘ftbl’] = ‘Name_of_the_target_DB_table’;
dbic[i][‘ftbl_nr’] = ‘Number_of_the_relevant_record_in_the_fileToTbl_table’;
dbic[i][‘pipe’] = 1; // ( or 0 1: file is part of an import pipeline, 0: only one file,no pipe)
dbic[i][‘loaded’] = 0; // ( or 1 if already loaded)
dbic[i][‘err’] = ‘error_text_or_error_code’;

“i” is an index counting the files according to the loading order defined in “fileToTbl”.

The array will be maintained in PHP’s $_SESSION and shall in addition be part of a JSON object transferred to the client as the Ajax answer of each client/server interaction phase until all files are processed. See again the previous article for the defined phases and steps.

Elements of a simple upload form

Let us now turn the the client and the key preparations there for initializing Phase I. We first build a very simple HTML 4 compatible form to trigger the upload job. We shall extend this web page area later on by progress bars. In the beginning our web page area for the upload shall look as simple as this:

In our project this Web UI area and its contained FORM tag are part of a PHP template (ITX or Smarty TPL) whose placeholder variables are filled by a PHP web page generator program. The TPL handling is assumed to be performed by a specific Template Control Object [TCO] which uses functionality of the chosen template engine. In our examples below we refer to the ITX case where placeholders of the form “{PLACE_HOLDER_NAME}” are to be filled. The template object shall be identified in our PHP code examples by a variable “tpl” of the TCO as $this->tpl.

In the following HTML code of our template we have used speaking IDs. We leave the simple CSS formatting to the reader. Concentrate instead on the FORM tag defined:

<!-- BEGIN UPLOAD -->											
<div id="div_upload_cont">
	<div id="div_upload">
		<div id="upl_float_cont">
			<div id="div_upl_header">
				<p id="upl_header"><span></span></p>
			</div>
			<div id="csv_file">
				<p><span class="fsnorm">CSV-files: </span><span id="num_open_files" class="bred">0</span></p>
			</div>
			<div id="imp_file">
				<p><span class="fsnorm">Imported-files: </span><span id="num_extracted_files" class="bgreen">0</span></p>
			</div>
			<p class="floatstopboth"> </p>
		</div>
		<form id="form_upload" name="init_file_form"  action="handle_uploaded_init_files.php5" method="POST" enctype="multipart/form-data" >

			<input type="hidden" name="{SESS_UPL_PROGR_FIELD_NAME}" id="hinp_progress_key_name" value="upl">
			
			<div id="file_cont">
				<input type="file" name="init_file" id="inp_upl_file" >
				<a id="but_submit_upl"  class="basic_but" href="#">Start Upload</a>
				<p class="float_stop"> </p>
			</div>
			
			<input type="hidden" name="upl_tbl_num" id="hinp_upl_tbl_num">
			<input type="hidden" name="upl_tbl_name" id="hinp_upl_tbl_name">
			<input type="hidden" name="upl_
tbl_snr" id="hinp_upl_tbl_snr">
			<input type="hidden" name="upl_file_succ" id="hinp_upl_succ" value="0">
			<input type="hidden" name="upl_file_name" id="hinp_upl_file_name" value="0">
			<input type="hidden" name="upl_file_pipe" id="hinp_upl_file_pipe" value="0">
			<input type="hidden" name="rt" id="hinp_upl_run_type">
		</form>
	</div>
</div>												
<!-- END UPLOAD -->

 
The “init” in some names is unimportant and project specific. In our present context you may assume it indicates Phase I.

The whole area of the enclosing DIV (id=”div_upload_cont”) is controlled on the JS side by an associated Control object “CtrlO_FileUpl” derived from a respective class (see below). The button defined above is used to trigger the file transfer process. This is done by a specific CtrlO method. Therefore, neither a form submit button is used, nor a special “href”-definition is required.

Note the variety of hidden input fields defined inside the FORM tag. Most of these fields will be used in several subsequent Ajax phases until the complete upload process is finalized.

However, during the starting Phase I only the file input field and the first hidden input field are relevant. The first and somewhat special input field can be identified in the code above by a private attribute: lbl=”progress”. (This attribute is otherwise unimportant). The “name” attribute of this input field is determined by a template placeholder which is replaced by the PHP TCO during web page creation.

	<input type="hidden" lbl="progress" name="{SESSION_UPLOAD_PROGRESS_NAME}" id="hinp_progress_name" value="upl">

 
Note: The position of the first special input was chosen on purpose. We deliver an explanation why the order of the data fields may become important during the POST transfer in a later section of this article.

I want to add a second note here, whose importance we shall understand later in this article series. As we have assumed the web page containing the upload form will be created by a PHP program with the help of a TCO. Now:

The PHP program creating the web page with the template based upload form must initiate a PHP session !

What is the (first) special input field used for?

Our first hidden input field must exist to trigger the initialization of the provision of progress data for the file transfer on the server. The server will look for the name and value of this input variable. The name has to follow rules so that the server recognizes it as special. The value will be used to define a key for accessing progress information in the $_SESSION array.

Actually, the name of the input field has to be identical to the value of the following PHP ini-variable:

; The index name (concatenated with the prefix) in $_SESSION
; containing the upload progress information
; Default Value: “PHP_SESSION_UPLOAD_PROGRESS”
; Development Value: “PHP_SESSION_UPLOAD_PROGRESS”
; Production Value: “PHP_SESSION_UPLOAD_PROGRESS”
; http://php.net/session.upload-progress.name
session.upload_progress.name = “PHP_SESSION_UPLOAD_PROGRESS”

This variable is defined in the php.ini file on the Apache server (normally it is located at “/etc/php5/apache2/php.ini”). It is essential that the server receives data associated with this name in the $_POST or $_GET array; otherwise the PHP engine will not care for the supply of upload progress information.

r
In a PHP code we can retrieve the value of the required name (from the variable settings in the PHP ini-file) by using the PHP function get_ini(). A PHP code snippet of the page generator TCO for filling our ITX-Block would look like:

 // Treatment of Upload Area 
$upload_block = "UPLOAD"; 
$tpl_hinp_upl_progr_field_name = "SESS_UPL_PROGR_FIELD_NAME"; 
$val_hinp_upl_progr_field_name = ini_get("session.upload_progress.name");
if ( $this->show_upload == 1 ) {
	$this->tpl->setCurrentBlock($upload_block);
		$this->tpl->setVariable($tpl_hinp_upl_progr_field_name, $val_hinp_upl_progr_field_name);
	$this->tpl->parseCurrentBlock();
}		 		

 
leading to

<input type="hidden" lbl="progress" name="PHP_SESSION_UPLOAD_PROGRESS" id="hinp_progress_key_name" value="upl">

 
inside the created web page. The structure of the code snippet would be very similar in case of the SMARTY engine.

Note: The defined value “upl” of the now named input field will later on be used to compose the key which we need to identify the array element containing progress information in the $_SESSION array.

The Javascript CtrlO for the Upload Area

As we have said, we control all event related and all Ajax action – e.g. of course in the initial phase I – by a defined CtrlO on the JS side. Such a singleton object responsible for our upload form may be derived from the following “class” definition:

GOC.CtrlO_FileUpl = new Ctrl_File_Upl('CtrlO_FileUpl');   

function Ctrl_File_Upl(my_name) {
	
	this.obj_name = "Obj_" + my_name;
	this.GOC = GOC; 
....
	// Timeout for file transfer process
	this.timeout = 100000; 

....
	// define selectors of the div and form 
		this.div_upload_cont_sel 	= "#" + "div_upload_cont";
		this.div_upload_sel 		= "#" + "div_upload";
		this.p_header_upload_sel 	= "#" + "upl_header" + " > span";
		this.form_upload_sel 		= "#" + "form_upload";
		this.input_file_sel 		= "#" + "inp_upl_file";
		this.upl_submit_but 		= "#" + "but_submit_upl";
		
		this.hinp_upl_tbl_num_sel	= "#" + "hinp_upl_tbl_num";			
		this.hinp_upl_tbl_name_sel	= "#" + "hinp_upl_tbl_name";			
		this.hinp_upl_tbl_snr_sel	= "#" + "hinp_upl_tbl_snr";			
		this.hinp_upl_succ_sel 		= "#" + "hinp_upl_succ";			
		this.hinp_upl_run_type_sel 	= "#" + "hinp_upl_run_type";			
		this.hinp_upl_file_name_sel 	= "#" + "hinp_upl_file_name";			
		this.hinp_upl_file_pipe_sel 	= "#" + "hinp_upl_file_pipe";			
		
		this.num_open_files_sel		= '#' + "num_open_files";
		this.num_extracted_files_sel	= '#' + "num_extracted_files";
...
	// Determine URL for the Form 
		this.url = $(this.form_upload_sel).attr('action'); 
		console.log("Form_Upload_file - url = " + this.url);  				

	// Register events with jQquery 
		this.register_form_events(); 
....
}

// Method to register events   
Ctrl_File_Upl.prototype.register_form_events = function() {
	// this indirectly also calls the secondly defined proxy method below 
	$(this.upl_submit_but).click(
		$.proxy(this, 'submit_form') 
	);
	// The real method called 	
	$(this.form_upload_sel).submit( 
		$.proxy(this, 'upl_file') 
	); 
}; 

 
We shall look at other methods of such an object later on. Let us first look a bit closer at some of the above definitions.

Side aspects: The GOC object indicated in the first
line is a special object which controls singleton objects in our JS code. I call such an object “Global Object Controller”. Thus we avoid placing the variety of our specific CtrlO objects into the global JS space. Note that such a GOC can also be used for dispatching knowledge about all created singleton CtrlO objects to all objects (e.g. each of the CtrlOs) which may need to know about their existence. Although maybe interesting in itself we do not look at the GOC in detail in this article series.

The variables defined in the beginning of our class definition will be used as jQuery selectors for several objects of our upload container DIV in the CtrlO methods defined below.

The URL of the target PHP program to be addressed by the Ajax request of Phase I is in our example read from the related attribute of the HTML form tag. See the HTML code above for it.

Important: Note the use of jQuery’s $.proxy-mechanism to encapsulate event control in methods of the CtrlO. The definitions given associate a specific event occuring at one of the HTML events with a CtrlO method.

Read more about using $.proxy in the jQuery documentation https://api.jquery.com/jQuery.proxy/. The “trick” here is to define the proper context for the JS “this“-operator used later in the triggered objects methods; the context has to be switched explicitly from the HTML element affected by the user event to the CtrlO object’s method. $.proxy does this for us in a simple, elegant way. You may read more about this trick in another article series of this blog
Fallen beim Statuscheck lang laufender PHP-Jobs mit Ajax – III

We shall look at the central method “upl_file” for starting the upload in a minute.

First obstacle: A method to transfer files and POST data at the same time via jQuery’s Ajax interface

Unfortunately, the transfer of file data from a standard form is not as simple via jQuery’s Ajax interface as soem kind readers may expect. E.g. you run into trouble, if you want to transfer normal input/textarea data from a form together with file (data map) data to a (PHP) server at the same time via the POST mechanism. This is due to the fact that standard settings for the Ajax interface of jQuery may not cover what is required both for file transfer and standard data transfer. Data from standard input elements must be processed to appear in the form of a query string, fitting to the default content-type “application/x-www-form-urlencoded”. However, the corresponding $.ajax settings do not work with file uploads. This is described in the following articles:
http://stackoverflow.com/questions/5392344/sending-multipart-formdata-with-jquery-ajax
http://abandon.ie/notebook/simple-file-uploads-using-jquery-ajax

What is required to overcome this problem? As described in the named articles a suitable step is to define an internal “FormData” object and attach the information gathered in the relevant input fields of our HTML FORM to this object. The data of the internal “FormData” object are then used in the Ajax controlled transfer with some special parametrization of the Ajax environment (more precise: of the XMLHttpRequest object).

In discussions with some JS developers used to conventional JS coding most find this approach more confusing than helpful. Actually, I personally find it elegant and fully in
line with my general attitude of using internal objects and their methods to control all aspects of user interaction, events and Ajax communication.

The resulting method “upl_file” to start the upload via Ajax looks as follows – note especially the creation of the FormData object :

Ctrl_File_Upl.prototype.submit_form = function (e) {
	e.preventDefault();
        $(this.form_upload_sel).submit(); 
};

Ctrl_File_Upl.prototype.upl_file = function(e) { 
	// Prevent Default action 
	e.preventDefault();
	
	// Set cursor to wait 
	$('body').css('cursor', 'wait' ); 

	// Some variables
	var form_data, url;
				
	// The identification key for the uploaded file in the server's $_Files
	var file_id_key = 'init_file';
				
	// Reset the values for the numbers of uploaded/open files 
	this.num_extracted_files = 0; 
	this.num_open_files = 0; 
	$(this.num_extracted_files_sel).html(this.num_extracted_files); 
	$(this.num_open_files_sel).html(this.num_open_files); 
....			
....			
	// Create a FormData object   
	form_data = new FormData();

	// Firstly (!!), add the hidden data to the DataForm object  
	var params = $(this.form_upload_sel).serializeArray();
	$.each(params, function (i, val) {
		form_data.append(val.name, val.value);
	});	

	// Secondly, fill the form with the information of the chosen file 
	// for the File API supported in present FF 
	$.each($(this.input_file_sel)[0].files, function(i, file) {
		if (i == 0) {
			form_data.append(file_id_key, file);
		}
	});
		/*	 
		// Multiple file selections in HTML 5 
		$.each($(this.input_file_sel)[0].files, function(key,value) {
			form_data.append(key, value); 
		});
		*/

	// "file" will be set and analyzed as a GET parameter to the PHP target url  
	url = this.url + "?file";
				
	// Time measures 
	this.date_start = new Date(); 
	this.ajax_transfer_start = this.date_start.getTime(); 
	console.log("From Ctrl_File_Upl.success_ajax_file_upl() ::  ajax_start = " + this.ajax_transfer_start);  

	// Setup Ajax 
	$.ajax({
		// contentType: "application/x-www-form-urlencoded; charset=ISO-8859-1",
		url: url, 
		context:  GOC[this.obj_name],
		timeout: 100000,
		data: form_data, 
		type: 'POST', 
		cache: false, 
		dataType: 'json', 
		contentType: false,
		processData: false, 
	
		error: this.error_ajax_file_upl,
		success: this.success_ajax_file_upl
	});
};	

 
Let us discuss some aspects of this method in detail:

• e.preventDefault is used to prevent that the event triggers a standard reaction of the affected HTML elements. The respective standard event capture and bubbling phases throughout the HTML element hierarchy are interrupted and only the defined CtrlO method code is executed.
• We define a key name “init_file” for our file to be able later on to identify its precisely in the PHP superglobal array $_FILES. This is not only done for convenience reasons: Although we only upload exactly one file in our present example based on HTML 4.1, we should be prepared to extend our methods to possible multi-file selection options of the modern HTML 5 file upload API.
• In the beginning of Phase I no files of the ZIP container were processed yet. Therefore, we set the numbers in the respective fields of our form to zero.
• We create the required internal “FormData”-object.
• Important:
  We append the special input field defining the key for upload progress information in the $_SESSION array first – i.e.
  before we append any file data.
   
  Please, do not ignore this point! It took me hours in the beginning to find out that a different order in the data transfer really leads to a complete failure of the whole concept of providing progress data in the $_SESSION superglobal ! Much later and by chance I found a related hint in one of the comments of PHP’s documentation http://php.net/manual/de/session.upload-progress.php

  The point is that the target PHP program of the (Ajax controlled) transfer process receives all data via the POST mechanism – but the PHP 5.4 engine has to recognize already in the very beginning of the transfer that a data upload whose progress shall be followed is initiated. And here an immediate filling of the relevant $_POST-array field is absolutely necessary before the file data appear in the POST buffer. I never had thought about whether there is a ordered sequence of information transfer during the POST process – but there is ! Data for the first fields of a form are transferred first! So, to be consistent always keep the special input field at the top of all other fields providing data in your HTLM FORM tag. In our case the POST data stream is actually derived from the elements of the internal FormData object – but there the same ordering rules are valid. Therefore, we append the data of this input field first to the FormData object.

• We retrieve the information of the HTML FORM by jQuery’s serialzeArray() functionality. The trick with

  params = $(this.form_upload_sel).serializeArray();

  is that the method $.serializeArray() of the jQuery object does not serialize input data for files or buttons of the HTML FORM tag. So, we only add the values of the hidden input arrays – and among these our special parameter.

• We now read the information about the file(s) selected in our HTML FORM – we do this already in form of a loop over all possible files of a HTML 5 multiselection field – although we do not really use multiple file selection in our example case. The selector variant in our code is valid for present Firefox browsers which supports a modern HTML5 file API (also in HTML 4.1 code). See e.g.:
  https://developer.mozilla.org/en-US/docs/Using_files_from_web_applications
  http://www.sanwebe.com/2013/10/check-input-file-size-before-submit-file-api-jquery
  http://stackoverflow.com/questions/5392344/sending-multipart-formdata-with-jquery-ajax

  Annotation: It would in our case also have worked without the first [0] as there are no more matching elements for the selector; using the [0] seems however to be good style … For MS IE you need probably version MS IE 10 or 11. I have not tested this.

• We supply a GET-parameter “file” to our url to explicitly distinguish Phase I from later phases – where we shall define another parameter.
• Eventually we set up our Ajax environment and trigger the Ajax communication via jQuery’s $.ajax() method.
   
  Important: Note again that we explicitly set the context for the this-operator of the Ajax interaction environment to our present CtrlO, which itself is an element of the mentioned GOC. Only by using this trick, we can be sure that the “this”-operator in the method used to handle the Ajax response later will
  refer to our present CtrlO. This is really important; otherwise the context would refer to the HTML object triggering the Ajax communication.
• Note also another the important setting: processData: false
  This prevents jQuery’s Ajax interface from changing data maps (as our file data) into the form of GET variables – thus making it possible to transfer the file data correctly via jQuery’s Ajax interface functions. By setting “contentType: false” we tell the Ajax interface in addition not to care for data types.
• Last but not least we define methods of our present CtrlO object “GOC.CtrlO_FileUpl” to be responsible for dealing with errors or the Ajax response object in case of a successful communication cycle of our Phase I. We shall look at these methods in a later article.

Note that according to our present setup the target PHP program addressed by the “url” will have to care about input data arriving in the following three superglobals:

$_GET, $_POST and of course $_FILES

Enough for today. We shall see what happens on the server side in the next article of this series to come. At least, we have set up everything such that the server can recognize at the beginning of the data transfer that the progress of the file data transmission shall be tracked.

Please, be a bit patient. The next 3 weeks i am involved in a different project. But the article

CSV file upload with Zip containers, jQuery, Ajax and PHP 5.4 status tracking – III

will be written.