CSV file upload with ZIP containers, jQuery, Ajax and PHP 5.4 progress tracking – II

In the first article of this series I summarized which phases and steps we may have to cover during an Ajax driven file upload procedure for a ZIP container file. See:

CSV file upload with ZIP containers, jQuery, Ajax and PHP 5.4 progress tracking – I

In this second article, I shall describe in more detail what has to be done during the first step identified for Phase I – the file transfer phase, where a ZIP container file is to be sent over the Internet from the client to the server. We look at the client side and describe some special settings and “tricks” which are required to enable a working and parallel transfer of the ZIP container file and other parameter data to the server via jQuery’s Ajax interface.

Note that the information given below should be regarded as a collection of suggestions. The code snippets are enhanced to make them understandable. Everything can of course be programmed in a different and more efficient way.

We start with some minor preparations to take care of:

Naming conventions for the CSV-files of the Zip container and a related database table

The server must get a chance to distinguish between the required actions for the handling of each of the CSV files in the transferred Zip container. We do this by setting up a small table “fileToTbl” in the LAMP server’s database. We use this table to associate a file name with a target table of the database. A file “alpha.csv” may such be associated with a DB table named “alpha”.

If there are several files referring to a split sequence of data for one and the same target database table the CSV file names may get an integer suffix like “alpha_1.csv, alpha_2.csv, …” – which indicates a loading sequence. The server can extract relevant parts of the file names by the PHP function “explode()” during the treatment of the transferred files.

The table “fileToTbl” may furthermore contain a column “onr” which via an integer defines the standard order by which all possibly received files should be imported into the database. Thus, typical columns would be:

nr, file_name, db_table_name, onr

It is the user’s responsibility to fill only CSV files with the defined names into the ZIP-container to be uploaded. If during ZIP file extraction files are detected which do not fulfill the naming conventions the server should issue a warning in its Ajax response.

Note that although the table “fileToTbl” includes information about all possible files and their names, the number of CSV files send in a specific upload process may be much lower than the maximum possible number. The user may want to update only some input data tables.

Control array for a database import pipeline

The previously described measure enables the server to build up an ordered “pipeline” for loading the received CSV files sequentially into their target tables in Phases II, III, …. When the target PHP program of Phase I server extracts the files from the transferred ZIP-container it can analyze the file names and build up an ordered sequence for the import according to the information found in the table “fileToTbl”.

The information about the loading sequence can be encoded into an array. We call this array “DB Import Control array”, short DBIC-array. It can and will be updated at the end of each Ajax phase and will also exchanged with the client. The (associative) BDIC array should contain information about which files have already been loaded, which files still are to be loaded
and if an error occured. Therefore, elements of the array could be:

dbic[i][‘fname’] = ‘Name_of_the_i-th_file’;
dbic[i][‘fsize’] = ‘Size_of_the_i-th_file’;
dbic[i][‘ftbl’] = ‘Name_of_the_target_DB_table’;
dbic[i][‘ftbl_nr’] = ‘Number_of_the_relevant_record_in_the_fileToTbl_table’;
dbic[i][‘pipe’] = 1; // ( or 0 1: file is part of an import pipeline, 0: only one file,no pipe)
dbic[i][‘loaded’] = 0; // ( or 1 if already loaded)
dbic[i][‘err’] = ‘error_text_or_error_code’;

“i” is an index counting the files according to the loading order defined in “fileToTbl”.

The array will be maintained in PHP’s $_SESSION and shall in addition be part of a JSON object transferred to the client as the Ajax answer of each client/server interaction phase until all files are processed. See again the previous article for the defined phases and steps.

Elements of a simple upload form

Let us now turn the the client and the key preparations there for initializing Phase I. We first build a very simple HTML 4 compatible form to trigger the upload job. We shall extend this web page area later on by progress bars. In the beginning our web page area for the upload shall look as simple as this:

upl_1

In our project this Web UI area and its contained FORM tag are part of a PHP template (ITX or Smarty TPL) whose placeholder variables are filled by a PHP web page generator program. The TPL handling is assumed to be performed by a specific Template Control Object [TCO] which uses functionality of the chosen template engine. In our examples below we refer to the ITX case where placeholders of the form “{PLACE_HOLDER_NAME}” are to be filled. The template object shall be identified in our PHP code examples by a variable “tpl” of the TCO as $this->tpl.

In the following HTML code of our template we have used speaking IDs. We leave the simple CSS formatting to the reader. Concentrate instead on the FORM tag defined:

<!-- BEGIN UPLOAD -->											
<div id="div_upload_cont">
	<div id="div_upload">
		<div id="upl_float_cont">
			<div id="div_upl_header">
				<p id="upl_header"><span></span></p>
			</div>
			<div id="csv_file">
				<p><span class="fsnorm">CSV-files: </span><span id="num_open_files" class="bred">0</span></p>
			</div>
			<div id="imp_file">
				<p><span class="fsnorm">Imported-files: </span><span id="num_extracted_files" class="bgreen">0</span></p>
			</div>
			<p class="floatstopboth"> </p>
		</div>
		<form id="form_upload" name="init_file_form"  action="handle_uploaded_init_files.php5" method="POST" enctype="multipart/form-data" >

			<input type="hidden" name="{SESS_UPL_PROGR_FIELD_NAME}" id="hinp_progress_key_name" value="upl">
			
			<div id="file_cont">
				<input type="file" name="init_file" id="inp_upl_file" >
				<a id="but_submit_upl"  class="basic_but" href="#">Start Upload</a>
				<p class="float_stop"> </p>
			</div>
			
			<input type="hidden" name="upl_tbl_num" id="hinp_upl_tbl_num">
			<input type="hidden" name="upl_tbl_name" id="hinp_upl_tbl_name">
			<input type="hidden" name="upl_
tbl_snr" id="hinp_upl_tbl_snr">
			<input type="hidden" name="upl_file_succ" id="hinp_upl_succ" value="0">
			<input type="hidden" name="upl_file_name" id="hinp_upl_file_name" value="0">
			<input type="hidden" name="upl_file_pipe" id="hinp_upl_file_pipe" value="0">
			<input type="hidden" name="rt" id="hinp_upl_run_type">
		</form>
	</div>
</div>												
<!-- END UPLOAD -->

 
The “init” in some names is unimportant and project specific. In our present context you may assume it indicates Phase I.

The whole area of the enclosing DIV (id=”div_upload_cont”) is controlled on the JS side by an associated Control object “CtrlO_FileUpl” derived from a respective class (see below). The button defined above is used to trigger the file transfer process. This is done by a specific CtrlO method. Therefore, neither a form submit button is used, nor a special “href”-definition is required.

Note the variety of hidden input fields defined inside the FORM tag. Most of these fields will be used in several subsequent Ajax phases until the complete upload process is finalized.

However, during the starting Phase I only the file input field and the first hidden input field are relevant. The first and somewhat special input field can be identified in the code above by a private attribute: lbl=”progress”. (This attribute is otherwise unimportant). The “name” attribute of this input field is determined by a template placeholder which is replaced by the PHP TCO during web page creation.

	<input type="hidden" lbl="progress" name="{SESSION_UPLOAD_PROGRESS_NAME}" id="hinp_progress_name" value="upl">

 
Note: The position of the first special input was chosen on purpose. We deliver an explanation why the order of the data fields may become important during the POST transfer in a later section of this article.

I want to add a second note here, whose importance we shall understand later in this article series. As we have assumed the web page containing the upload form will be created by a PHP program with the help of a TCO. Now:

The PHP program creating the web page with the template based upload form must initiate a PHP session !

What is the (first) special input field used for?

Our first hidden input field must exist to trigger the initialization of the provision of progress data for the file transfer on the server. The server will look for the name and value of this input variable. The name has to follow rules so that the server recognizes it as special. The value will be used to define a key for accessing progress information in the $_SESSION array.

Actually, the name of the input field has to be identical to the value of the following PHP ini-variable:

; The index name (concatenated with the prefix) in $_SESSION
; containing the upload progress information
; Default Value: “PHP_SESSION_UPLOAD_PROGRESS”
; Development Value: “PHP_SESSION_UPLOAD_PROGRESS”
; Production Value: “PHP_SESSION_UPLOAD_PROGRESS”
; http://php.net/session.upload-progress.name
session.upload_progress.name = “PHP_SESSION_UPLOAD_PROGRESS”

This variable is defined in the php.ini file on the Apache server (normally it is located at “/etc/php5/apache2/php.ini”). It is essential that the server receives data associated with this name in the $_POST or $_GET array; otherwise the PHP engine will not care for the supply of upload progress information.

r
In a PHP code we can retrieve the value of the required name (from the variable settings in the PHP ini-file) by using the PHP function get_ini(). A PHP code snippet of the page generator TCO for filling our ITX-Block would look like:

 // Treatment of Upload Area 
$upload_block = "UPLOAD"; 
$tpl_hinp_upl_progr_field_name = "SESS_UPL_PROGR_FIELD_NAME"; 
$val_hinp_upl_progr_field_name = ini_get("session.upload_progress.name");
if ( $this->show_upload == 1 ) {
	$this->tpl->setCurrentBlock($upload_block);
		$this->tpl->setVariable($tpl_hinp_upl_progr_field_name, $val_hinp_upl_progr_field_name);
	$this->tpl->parseCurrentBlock();
}		 		

 
leading to

<input type="hidden" lbl="progress" name="PHP_SESSION_UPLOAD_PROGRESS" id="hinp_progress_key_name" value="upl">

 
inside the created web page. The structure of the code snippet would be very similar in case of the SMARTY engine.

Note: The defined value “upl” of the now named input field will later on be used to compose the key which we need to identify the array element containing progress information in the $_SESSION array.

The Javascript CtrlO for the Upload Area

As we have said, we control all event related and all Ajax action – e.g. of course in the initial phase I – by a defined CtrlO on the JS side. Such a singleton object responsible for our upload form may be derived from the following “class” definition:

GOC.CtrlO_FileUpl = new Ctrl_File_Upl('CtrlO_FileUpl');   

function Ctrl_File_Upl(my_name) {
	
	this.obj_name = "Obj_" + my_name;
	this.GOC = GOC; 
....
	// Timeout for file transfer process
	this.timeout = 100000; 

....
	// define selectors of the div and form 
		this.div_upload_cont_sel 	= "#" + "div_upload_cont";
		this.div_upload_sel 		= "#" + "div_upload";
		this.p_header_upload_sel 	= "#" + "upl_header" + " > span";
		this.form_upload_sel 		= "#" + "form_upload";
		this.input_file_sel 		= "#" + "inp_upl_file";
		this.upl_submit_but 		= "#" + "but_submit_upl";
		
		this.hinp_upl_tbl_num_sel	= "#" + "hinp_upl_tbl_num";			
		this.hinp_upl_tbl_name_sel	= "#" + "hinp_upl_tbl_name";			
		this.hinp_upl_tbl_snr_sel	= "#" + "hinp_upl_tbl_snr";			
		this.hinp_upl_succ_sel 		= "#" + "hinp_upl_succ";			
		this.hinp_upl_run_type_sel 	= "#" + "hinp_upl_run_type";			
		this.hinp_upl_file_name_sel 	= "#" + "hinp_upl_file_name";			
		this.hinp_upl_file_pipe_sel 	= "#" + "hinp_upl_file_pipe";			
		
		this.num_open_files_sel		= '#' + "num_open_files";
		this.num_extracted_files_sel	= '#' + "num_extracted_files";
...
	// Determine URL for the Form 
		this.url = $(this.form_upload_sel).attr('action'); 
		console.log("Form_Upload_file - url = " + this.url);  				

	// Register events with jQquery 
		this.register_form_events(); 
....
}

// Method to register events   
Ctrl_File_Upl.prototype.register_form_events = function() {
	// this indirectly also calls the secondly defined proxy method below 
	$(this.upl_submit_but).click(
		$.proxy(this, 'submit_form') 
	);
	// The real method called 	
	$(this.form_upload_sel).submit( 
		$.proxy(this, 'upl_file') 
	); 
}; 

 
We shall look at other methods of such an object later on. Let us first look a bit closer at some of the above definitions.

Side aspects: The GOC object indicated in the first
line is a special object which controls singleton objects in our JS code. I call such an object “Global Object Controller”. Thus we avoid placing the variety of our specific CtrlO objects into the global JS space. Note that such a GOC can also be used for dispatching knowledge about all created singleton CtrlO objects to all objects (e.g. each of the CtrlOs) which may need to know about their existence. Although maybe interesting in itself we do not look at the GOC in detail in this article series.

The variables defined in the beginning of our class definition will be used as jQuery selectors for several objects of our upload container DIV in the CtrlO methods defined below.

The URL of the target PHP program to be addressed by the Ajax request of Phase I is in our example read from the related attribute of the HTML form tag. See the HTML code above for it.

Important: Note the use of jQuery’s $.proxy-mechanism to encapsulate event control in methods of the CtrlO. The definitions given associate a specific event occuring at one of the HTML events with a CtrlO method.

Read more about using $.proxy in the jQuery documentation https://api.jquery.com/jQuery.proxy/. The “trick” here is to define the proper context for the JS “this“-operator used later in the triggered objects methods; the context has to be switched explicitly from the HTML element affected by the user event to the CtrlO object’s method. $.proxy does this for us in a simple, elegant way. You may read more about this trick in another article series of this blog
Fallen beim Statuscheck lang laufender PHP-Jobs mit Ajax – III

We shall look at the central method “upl_file” for starting the upload in a minute.

First obstacle: A method to transfer files and POST data at the same time via jQuery’s Ajax interface

Unfortunately, the transfer of file data from a standard form is not as simple via jQuery’s Ajax interface as soem kind readers may expect. E.g. you run into trouble, if you want to transfer normal input/textarea data from a form together with file (data map) data to a (PHP) server at the same time via the POST mechanism. This is due to the fact that standard settings for the Ajax interface of jQuery may not cover what is required both for file transfer and standard data transfer. Data from standard input elements must be processed to appear in the form of a query string, fitting to the default content-type “application/x-www-form-urlencoded”. However, the corresponding $.ajax settings do not work with file uploads. This is described in the following articles:
http://stackoverflow.com/questions/5392344/sending-multipart-formdata-with-jquery-ajax
http://abandon.ie/notebook/simple-file-uploads-using-jquery-ajax

What is required to overcome this problem? As described in the named articles a suitable step is to define an internal “FormData” object and attach the information gathered in the relevant input fields of our HTML FORM to this object. The data of the internal “FormData” object are then used in the Ajax controlled transfer with some special parametrization of the Ajax environment (more precise: of the XMLHttpRequest object).

In discussions with some JS developers used to conventional JS coding most find this approach more confusing than helpful. Actually, I personally find it elegant and fully in
line with my general attitude of using internal objects and their methods to control all aspects of user interaction, events and Ajax communication.

The resulting method “upl_file” to start the upload via Ajax looks as follows – note especially the creation of the FormData object :

Ctrl_File_Upl.prototype.submit_form = function (e) {
	e.preventDefault();
        $(this.form_upload_sel).submit(); 
};

Ctrl_File_Upl.prototype.upl_file = function(e) { 
	// Prevent Default action 
	e.preventDefault();
	
	// Set cursor to wait 
	$('body').css('cursor', 'wait' ); 

	// Some variables
	var form_data, url;
				
	// The identification key for the uploaded file in the server's $_Files
	var file_id_key = 'init_file';
				
	// Reset the values for the numbers of uploaded/open files 
	this.num_extracted_files = 0; 
	this.num_open_files = 0; 
	$(this.num_extracted_files_sel).html(this.num_extracted_files); 
	$(this.num_open_files_sel).html(this.num_open_files); 
....			
....			
	// Create a FormData object   
	form_data = new FormData();

	// Firstly (!!), add the hidden data to the DataForm object  
	var params = $(this.form_upload_sel).serializeArray();
	$.each(params, function (i, val) {
		form_data.append(val.name, val.value);
	});	

	// Secondly, fill the form with the information of the chosen file 
	// for the File API supported in present FF 
	$.each($(this.input_file_sel)[0].files, function(i, file) {
		if (i == 0) {
			form_data.append(file_id_key, file);
		}
	});
		/*	 
		// Multiple file selections in HTML 5 
		$.each($(this.input_file_sel)[0].files, function(key,value) {
			form_data.append(key, value); 
		});
		*/

	// "file" will be set and analyzed as a GET parameter to the PHP target url  
	url = this.url + "?file";
				
	// Time measures 
	this.date_start = new Date(); 
	this.ajax_transfer_start = this.date_start.getTime(); 
	console.log("From Ctrl_File_Upl.success_ajax_file_upl() ::  ajax_start = " + this.ajax_transfer_start);  

	// Setup Ajax 
	$.ajax({
		// contentType: "application/x-www-form-urlencoded; charset=ISO-8859-1",
		url: url, 
		context:  GOC[this.obj_name],
		timeout: 100000,
		data: form_data, 
		type: 'POST', 
		cache: false, 
		dataType: 'json', 
		contentType: false,
		processData: false, 
	
		error: this.error_ajax_file_upl,
		success: this.success_ajax_file_upl
	});
};	

 
Let us discuss some aspects of this method in detail:

  • e.preventDefault is used to prevent that the event triggers a standard reaction of the affected HTML elements. The respective standard event capture and bubbling phases throughout the HTML element hierarchy are interrupted and only the defined CtrlO method code is executed.
  • We define a key name “init_file” for our file to be able later on to identify its precisely in the PHP superglobal array $_FILES. This is not only done for convenience reasons: Although we only upload exactly one file in our present example based on HTML 4.1, we should be prepared to extend our methods to possible multi-file selection options of the modern HTML 5 file upload API.
  • In the beginning of Phase I no files of the ZIP container were processed yet. Therefore, we set the numbers in the respective fields of our form to zero.
  • We create the required internal “FormData”-object.
  • Important:
    We append the special input field defining the key for upload progress information in the $_SESSION array first – i.e.
    before we append any file data.
     
    Please, do not ignore this point! It took me hours in the beginning to find out that a different order in the data transfer really leads to a complete failure of the whole concept of providing progress data in the $_SESSION superglobal ! Much later and by chance I found a related hint in one of the comments of PHP’s documentation http://php.net/manual/de/session.upload-progress.php

    The point is that the target PHP program of the (Ajax controlled) transfer process receives all data via the POST mechanism – but the PHP 5.4 engine has to recognize already in the very beginning of the transfer that a data upload whose progress shall be followed is initiated. And here an immediate filling of the relevant $_POST-array field is absolutely necessary before the file data appear in the POST buffer. I never had thought about whether there is a ordered sequence of information transfer during the POST process – but there is ! Data for the first fields of a form are transferred first! So, to be consistent always keep the special input field at the top of all other fields providing data in your HTLM FORM tag. In our case the POST data stream is actually derived from the elements of the internal FormData object – but there the same ordering rules are valid. Therefore, we append the data of this input field first to the FormData object.

  • We retrieve the information of the HTML FORM by jQuery’s serialzeArray() functionality. The trick with

    params = $(this.form_upload_sel).serializeArray();

    is that the method $.serializeArray() of the jQuery object does not serialize input data for files or buttons of the HTML FORM tag. So, we only add the values of the hidden input arrays – and among these our special parameter.

  • We now read the information about the file(s) selected in our HTML FORM – we do this already in form of a loop over all possible files of a HTML 5 multiselection field – although we do not really use multiple file selection in our example case. The selector variant in our code is valid for present Firefox browsers which supports a modern HTML5 file API (also in HTML 4.1 code). See e.g.:
    https://developer.mozilla.org/en-US/docs/Using_files_from_web_applications
    http://www.sanwebe.com/2013/10/check-input-file-size-before-submit-file-api-jquery
    http://stackoverflow.com/questions/5392344/sending-multipart-formdata-with-jquery-ajax

    Annotation: It would in our case also have worked without the first [0] as there are no more matching elements for the selector; using the [0] seems however to be good style … For MS IE you need probably version MS IE 10 or 11. I have not tested this.

  • We supply a GET-parameter “file” to our url to explicitly distinguish Phase I from later phases – where we shall define another parameter.
  • Eventually we set up our Ajax environment and trigger the Ajax communication via jQuery’s $.ajax() method.
     
    Important: Note again that we explicitly set the context for the this-operator of the Ajax interaction environment to our present CtrlO, which itself is an element of the mentioned GOC. Only by using this trick, we can be sure that the “this”-operator in the method used to handle the Ajax response later will
    refer to our present CtrlO. This is really important; otherwise the context would refer to the HTML object triggering the Ajax communication.
  • Note also another the important setting: processData: false
    This prevents jQuery’s Ajax interface from changing data maps (as our file data) into the form of GET variables – thus making it possible to transfer the file data correctly via jQuery’s Ajax interface functions. By setting “contentType: false” we tell the Ajax interface in addition not to care for data types.
  • Last but not least we define methods of our present CtrlO object “GOC.CtrlO_FileUpl” to be responsible for dealing with errors or the Ajax response object in case of a successful communication cycle of our Phase I. We shall look at these methods in a later article.

Note that according to our present setup the target PHP program addressed by the “url” will have to care about input data arriving in the following three superglobals:

$_GET, $_POST and of course $_FILES

Enough for today. We shall see what happens on the server side in the next article of this series to come. At least, we have set up everything such that the server can recognize at the beginning of the data transfer that the progress of the file data transmission shall be tracked.

Please, be a bit patient. The next 3 weeks i am involved in a different project. But the article

CSV file upload with Zip containers, jQuery, Ajax and PHP 5.4 status tracking – III

will be written.

CSV file upload with ZIP containers, jQuery, Ajax and PHP 5.4 progress tracking – I

This article series is written in support for French colleagues in a PHP collaboration project and therefore in English. I want to describe some basic elements of an

Ajax controlled file upload process between

  • a browser based User Interface (HTML4/5, Javascript, jQuery)
  • and some PHP/MySQL application programs on a LAMP server.

Our customer’s project depends on a periodic transfer of up to 40 different CSV files with a lot of input data (around 0.5 GByte) to a database server. A requirement of our customer was that the data transfer should be performed with a ZIP file as a container for the individual CSV data files. After the transfer the contents of the individual CSV files should be imported into specific tables of a database.

As our whole interface of the web application is Ajax based, we decided to control all transfers via jQuery’s Ajax API. Meanwhile, there are jQuery Plugins available for this type of task. However, we wanted to fully control all important phases of the file transfer and the data import – both on the client (browser) as well as on the server. This meant that we needed to program all basic steps during an Ajax communication cycle between the browser client and the LAMP server by ourselves. In addition we needed to guarantee some error control.

Personally, I found it a bit astonishing that such a seemingly simple task lead me to some relatively intricate obstacles to overcome. Although most of the necessary ingredients are documented on the Internet, the documentation is sparse and distributed. My objective with this article series is to provide a coherent picture of process design aspects, some coding tricks and also limitations of such a process. This may be useful also for other developers having to solve similar problems. However, if you want to read about a most simple and problem free approach to file upload tasks with Ajax you are probably looking at the wrong article.

Objectives and Assumptions

  • We want to upload several CSV-files (up to 40), whose contents shall be transferred to specific database tables.
  • These files shall be sent to the server in one ZIP file. Reasons for using a ZIP-container file: compression; limitations of the HTML4 file upload API.
  • As the Zip-container may get relatively large we want to see and control the transfer progress over the Internet by some means of PHP 5.4 – as far as possible today.
  • The server shall extract the files from the ZIP and build up a “pipeline” of these files for a subsequent database import of their contents.
  • The data import into database tables shall be done by a sequence of Ajax controlled PHP jobs. Reason: Intermediate information transfer to the client with the option to stop further processing.
  • The server shall decide by some naming conventions what to do with each file.
  • All steps shall be Ajax controlled – a relatively continuous flow of information between client and server has to be established.
  • For the sake of simplicity each Ajax answer of the server at the end of each controlled Ajax transaction cycle shall be encoded in form of a JSON object. (So, if you want to be particularly precise: we use Ajaj instead of Ajax.)

Wording used in this article series

  • JS, jQuery, Ajax:JS below stands for Javascript (on the
    client side). We furthermore use jQuery and its Ajax interface functionality. We expect JSON responses from the server. Although not completely correct we nevertheless use Ajax and Ajaj as synonyms in the articles of this series.
  • Upload: By “upload” we normally mean the whole process. It comprises a “file transfer process” from the Web client PC to the server and subsequent “database import processes”. However, sometimes and for reasons of simplicity we also use the expression “file upload” in a restricted sense – namely for the file transfer to the server, only. It should become clear from the context what we mean.
  • Main PHP program/job:
    The PHP program receiving and working with the transferred Zip-container file and its contents is called “main PHP program” or “main PHP job”. It has to be distinguished from “polling jobs” (see below).
  • Polling jobs: A sequence of additional PHP “polling jobs” may be triggered by the client. This is done in form of a time loop with a short period. A “polling job” on the LAMP server reads some status information of a previously started and still running “Main PHP job” (as e.g. the file transfer job or long lasting database import jobs). The status information of the running main job is fetched from a common data source as the $_SESSION or a database table accessible to both the status writing “main PHP job” and the status reading “polling job”. Each short timed “polling job” fulfills its own complete Ajax transaction cycle. The evaluation of the Ajax response triggers the next polling job if the main PHP job is still running. We come back to the concept of “Ajax driven polling jobs” later on.
  • CtrlOs: Each user interaction area of a web page – e.g. a HTML FORM in a DIV container – shall be completely controlled by a so called JS “Control object” [CtrlO]. A CtrlO encapsulates all reactions of the UI to events in well defined prototype methods. A CtrlO uses jQuery’s proxy mechanism to register events and delegate event handling to defined CtrlO methods. CtrlO methods furthermore control the Ajax communication with the server.
  • Phases: “Phases” describe a full cycle of defined Ajax interaction between client and server. An example of such a full cycle would be:

    HTML Form => Ajax Setup via JS CtrlO method => Submit via JS CtrlO method => POST/FILE data transfer => Server action (PHP) => JSON object as Ajax response => Client analysis of the JSON object via CtrlO method for the Ajax response

  • Client: The client is in our case typically a browser (Firefox) with active JS and jQuery. We do not care about specific requirements of MS IE browsers in this article series; but we assume that at least MS IE browsers > 10 should work.
  • Pipeline: The ordered sequence by which the files of the transferred ZIP container are imported into their related database.

Relevant phases

To get a more detailed overview over what is to be done we distinguish the following main phases and steps (I omit error handling in this overview which may occur at every step):

Phase I – file transfer, progress control and Zip extraction

Step I.1 – Client: Use a HTML form to choose a ZIP file (<input type=”file”>) and use methods of a specifically designed JS
Control object [CtrlO] to control subsequent actions on the client. Add parameter data (hidden input fields) and prepare an Ajax transaction for the file upload (=transfer) process.

Step I.2 – Client: Start the transfer the ZIP file over the Internet to the server. Submit a special parameter in addition to the file to trigger the provision of transfer progress information on the server. Prepare and start the Ajax communication and the data transfer by a CtrlO method.

Step I.3 – Server: Initialize the progress measurement and provide progress data in the $_SESSION array.

Step I.4 – Client/Server: Initiate a sequence of Ajax polling jobs via a JS time loop for reading the progress information on the server. Handle the Ajax response of each polling job in separate defined methods of a special CtrlO. React to error situations and stop the polling job time loop in case of errors or when the file transfer has finalized.

Step I.5 – Server: Extract, expand and save the CSV files from the Zip-container into a special upload folder on the server. This is done by using standard methods of the PHP ZIP class. Define/Suggest a sequence of imports of the data contents of the different files into file specific database tables. This defined sequence may be controlled via an array (“DB Import Control array” = DBIC-array ) which is kept and updated in the $_SESSION array on the server AND which is also sent back via a JSON object to the client.

Step I.6 – Server: Prepare and send an Ajax response in form of a JSON object to the client with affirmation messages about which CSV files have been received, the name of the files and the order in which they shall be processed. Include error messages and system messages if necessary. The JSON object shall contain the “DBIC”-array.

Step I.7 – Client: Analyze the Ajax response. Display success and error information. Display the number and name of files to be processed afterwards. Stop the time loop for polling jobs.

Phase II – Database import of a file in the pipeline

Step II.1 – Client: Prepare and start a new Ajax job with some parameters. The PHP target program of this job shall import the data of one of the already transferred CSV files. Among other things it should be defined, which file shall be processed (= imported into its associated database table) next. This parameter can follow the suggested order of the array which came from the server at the end of Phase I. All parameters can be set up in a separate (hidden) form with hidden input fields. Submit the Ajax job.

Step II.2 – Server: Start the database import on the server with a flexible PHP program. For small and medium sized files (up to approx. below 500000 lines) do it line by line by appropriate special PHP standard methods for handling CSV files. Check the data of each line where reasonable. Gather at least 20 lines in one INSERT statement to accelerate the import process. Write intermediate progress information into a $_SESSION array or a special database table. (This status information may be read by “polling jobs” started on the client.)
For huge files you may extend the import methods later on by using the special MySQL
command “LOAD DATA INFILE”.

Step II.3 – Client: Launch a sequence of status information polling jobs via a time loop. Handle the return information of each job in separate methods.

Step II.3 – Server: After a successful import of a defined file remove the file from its upload directory (delete it or move it somewhere else, e.g. in a history directory for uploads). Update you Control Array for the sequence of uploads with the following info: Which of the original files have been loaded? Which had errors? Which are still unprocessed? Prepare a JSON object for an Ajax respond (including the upload Control Array). Send it back to the client.

Step II.4 – Client: Analyze the server’s response. Stop any polling jobs issued after the previous submit. Continue with displaying information of the success of the database import of the handled file. Determine the next file to load. Continue with the elements of Step 5 described above.

Phases III + n – Client/Server – Loop:

Cycle through a sequence of Steps described under II.1 to II.4 for further phases until all files are processed or an error has occurred.

Enough for today. In the next article

CSV file upload with Zip containers, jQuery, Ajax and PHP 5.4 progress tracking – II

we shall cover some major elements of Phase I.

Geschwindigkeit/Übertragungsrate eines Netzwerkinterfaces unter Linux reduzieren

Im Rahmen aktueller Arbeiten an Ajax getriebenen Dateiuploads tauchte das Problem auf, mal kurzfristig die Geschwindigkeit einer Netzwerkverbindung drosseln zu müssen, um im lokalen Netz einer länger andauernden Dateitransfer zwischen einem Browser und einem Server zu simulieren.

Aus alten Zeiten hatte ich Werkzeuge wie “wondershaper” oder “trickle” im Kopf. Siehe etwa
http://www.ubuntugeek.com/use-bandwidth-shapers-wondershaper-or-trickle-to-limit-internet-connection-speed.html
http://askubuntu.com/questions/20872/how-do-i-limit-internet-bandwidth
http://unix.stackexchange.com/questions/83888/limit-outgoing-bandwidth-on-an-specific-interface
Diese Tools sind aber sehr alt. Entsprechende Pakete für “trickle” findet man für aktuelle Opensuse Versionen nicht mehr in den Standard-Repositories.

Eine einfach schnelle Lösung für Ethernet-Schnittstellen bietet das CLI Werkzeug “ethtool”, das zum Standardumfang der meisten Distributionen gehört. “ethtool” ist ein mächtiges Werkzeug zur Analyse und zur Konfiguration/Manipulation von Schnittstellen. Es lohnt sich, sich damit ein wenig zu beschäftigen. Zu meiner Schande muss ich gestehen, dass ich dies bislang eher versäumt habe.

Für unseren Anwendungszweck ist die Syntax aber sehr einfach. Heißt unser Ethernet Device etwa “enp8s0”, so erhalten wir einen Überblick über technische Daten der Schnittstelle mit :

mytux:~ # ethtool enp8s0
Settings for enp8s0:
        Supported ports: [ TP MII ]
        Supported link modes:   10baseT/Half 10baseT/Full 
                                100baseT/Half 100baseT/Full 
                                1000baseT/Half 1000baseT/Full 
        Supported pause frame use: No
        Supports auto-negotiation: Yes
        Advertised link modes:  10baseT/Half 10baseT/Full 
                                100baseT/Half 100baseT/Full 
                                1000baseT/Half 1000baseT/Full 
        Advertised pause frame use: Symmetric Receive-only
        Advertised auto-negotiation: Yes
        Link partner advertised link modes:  10baseT/Half 10baseT/Full 
                                             100baseT/Half 100baseT/Full 
                                             1000baseT/Full 
        Link partner advertised pause frame use: Symmetric Receive-only
        Link partner advertised auto-negotiation: Yes
        Speed: 1000Mb/s
        Duplex: Full
        Port: MII
        PHYAD: 0
        Transceiver: internal
        Auto-negotiation: on
        Supports Wake-on: pumbg
        Wake-on: g
        Current message level: 0x00000033 (51)
                               drv probe ifdown ifup
        Link detected: yes

Welche Netzwerkschnittstellen auf dem eigenen System verfügbar sind, ermittelt man natürlich mit
“ip link show” oder “ifconfig”:

mytux:~ # ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: enp8s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
    link/ether xx:xx:xx:xx:xx:xx brd ff:ff:ff:ff:ff:ff
3: enp9s0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether yy:yy:yy:yy:yy:yy brd ff:ff:ff:ff:ff:ff
4: vmnet1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 
1500 qdisc pfifo_fast state UNKNOWN mode DEFAULT group default qlen 1000
    link/ether nn:nn:nn:nn:nn:nn brd ff:ff:ff:ff:ff:ff
5: vmnet8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN mode DEFAULT group default qlen 1000
    link/ether mm:mm:mm:mm:mm:mm brd ff:ff:ff:ff:ff:ff

Die generelle Syntax zur Änderung der Geschwindigkeit mit ethtool ist:

ethtool -s DEVICE speed [SPEED] duplex [DUPLEX]

DEVICE ist durch den Schnittstellennamen, den man über ifconfig ermitteln kann, zu ersetzen. Welche SPEED Werte angegeben werden können, zeigt der Output von “ethtool DEVICE”; s. das Beispiel oben.

Will ich also die Übertragungsrate meiner Netzwerkschnittstelle temporär auf mögliche 10Mb/s reduzieren, so hilft:

mytux:~ # ethtool -s enp8s0 speed 10 duplex full

Und tatsächlich:

mytux:~ # ethtool enp8s0
Settings for enp8s0:
....    
	Link partner advertised auto-negotiation: Yes
        Speed: 10Mb/s
        Duplex: Full
....

Zurücksetzen erfolgt mit

mytux:~ # ethtool -s enp8s0 speed 1000 duplex full

Autonegotiation für weiterreichende Experimente, bei denen man Verbindungskabel abzieht/ansteckt, stoppt man übrigens mit

ethtool -s DEVICE autoneg off

Der Nachteil von “ethtool” gegenüber dem alten “trickle” ist übrigens, dass “trickle” durch Manipulationen im Userspace die Transfergeschwindigkeit für eine bestimmte Anwendung drosseln kann. Das geht mit ethtool nicht.